Nov 20th 2012, 17:03:06
Maybe some of those ROUND()s should be TRUNC()s. Simple explanation.
Or just simple computer 32-bit floating point errors. 32-bit floats are only accurate to 6 significant digits, so if you have billions of cash, it can be off by 1 when you do division if the 64-bit integer cash is converted into a 32-bit float before the division happens. (Solution, use 64-bit doubles)
For example, 11,234,567,890 dollars is converted to 1.123457+e10 and you lose 7890 dollars in the NW calculation.
I'll only be concerned if the formula is off by more than 2.
(The above is just conjecture, the code is more likely to be done in integer division, which is why there are TRUNC()s - the fractional parts are always dropped in integer division so the answer is always an integer).
Or just simple computer 32-bit floating point errors. 32-bit floats are only accurate to 6 significant digits, so if you have billions of cash, it can be off by 1 when you do division if the 64-bit integer cash is converted into a 32-bit float before the division happens. (Solution, use 64-bit doubles)
For example, 11,234,567,890 dollars is converted to 1.123457+e10 and you lose 7890 dollars in the NW calculation.
I'll only be concerned if the formula is off by more than 2.
(The above is just conjecture, the code is more likely to be done in integer division, which is why there are TRUNC()s - the fractional parts are always dropped in integer division so the answer is always an integer).