Re: This calculation is just wrong / computer can't count!

From:

"David Ching" <dc@remove-this.dcsoft.com>

Newsgroups:

microsoft.public.vc.mfc

Date:

Wed, 10 Oct 2007 09:44:53 -0700

Message-ID:

<Ll7Pi.4911$oA2.3753@nlpi068.nbdc.sbc.com>

"GT" <ContactGT_remove_@hotmail.com> wrote in message
news:017d7b1d$0$20741$c3e8da3@news.astraweb.com...

This is the problem - the calculations aren't working properly and the
displayed answer to any number of decimal places, even no decimal places
is wrong.

Now I'm confused. I thought your point was that the result was
0.333333333...7 <--- a 7 is the last digit, not a 3), in that case
using 3 decimal digits, 0.333 is certainly accurate and good enough for
you.

I'm not displaying this number - it is used at the first stage in a long
series of calculations. The number that originally raised this question
was a display of -0.0, which for a project management tool to display on
screen is very poor. I stepped into my code and saw the 7 on the end of a
number and tracked it and watched it propagate and start to effect the
calculations later on. Obviously, I have long since adjusted the display
code to prevent this from getting to the screen, but have been searching
for the 'right' way to do it in this thread.

Joe has already answered your concern about intermediate calculations. The
7 shown in the intermediate result means that it is stored internally with
full binary precision, which is exactly what you want in an intermediate
result. So that is not affecting the end calculation. -0.0 is technically
equivalent to 0.0, so the fact it showed this way prior to you fixing the
display does not indicate any bad calculations.

-- David