Precision of HP 30S

[Markus Sigg]

Hi,

I read that the 30S works with binary representations of numbers, whereas most other pocket calculators use decimal numbers. And I read that its precision is equivalent to 24 decimal places.

Does anybody have an explanation for following results?

(a) 1E10 + 0.11 - 1E10 gives 0.11
(b) 1E10 + 0.01 - 1E10 gives 0

Instead of 0.11 in (a) you can use 0.12345678 also and still get the correct result. So what's happening in (b)???

Things like these make it rather difficult to rely on the precision of this calculator or to estimate error bounds.

Markus

[hugh steers]

the hp30s and 9g models do, in fact, work in binary. they also, generally get the correct answers. but there are drawbacks to binary requiring the extra internal precision and also some devious value suppression going on.

its the supression that's expressing the behaviour you've seen. i wrote some stuff about working in binary here:

http://www.voidware.com/binarycalculators.htm

[Markus Sigg]

I think a rounding scheme like that makes the 24 internal digits somewhat useless. I don't see a rational justification for supressing x=0.01, x=0.02, and x=0.03 when calculating 1E10+x but keeping all digits of x=0.11 or even of x=0.12345678.

Are you able to explain the mechanism a bit more detailed? It seems that 1E10+0.01 is changed to 1E10, but 1E10+0.11 is kept. What's the exact rule? Suppose the result of an operation is a. What does the calculator do with this number? Your rounding algorithm needs a second number b to check whether a~b. How does it find b? It seems to me that the decimal expansion of the numbers is used here. Doesn't this imply that a decimal number representation is used?

Markus

[hugh steers]

its because 0.11 is roughly ten times bigger than 0.01.

here’s how i think it works (really i’m guessing). it doesn’t just use the final answer `a’. it compares `a’ to one of the terms of the input ‘b’. this suppression only applies to ADD and SUB at the top level and not to adds and subs used internally.

define ADD(x, y) to be 0 if |(x + y)/x| < eps, where eps = 1e-12, say.
and ADD(x,y) to be x + y, otherwise.

define SUB(x, y) to be 0 if |(x - y)/x| < eps, where eps = 1e-12, say.
and SUB(x,y) to be x - y, otherwise.

in the above “+” means the result of the floating point binary add (ie not true real number addition).

so, in your examples. ex1: 1e10 + 0.11 - 1e10 = 0.11
we have SUB(ADD(1e10,0.11), 1e10) = SUB(10000000000.11,1e10) = 0.11 because
(10000000000.11-1e10)/10000000000.11 = 1.0999999..e-11 > eps

but, ex2: 1e10 + 0.01 - 1e10 = 0 because
SUB(ADD(1e10,0.01),1e10) = SUB(10000000000.01,1e10) = 0, since
(10000000000.01-1e10)/10000000000.01 = 9.9999999..e-13 < eps

[Markus Sigg]

Your algorithm seems to be correct, thanks. Maybe eps=2e-12 is actually used.

My first thought was that certainly instead of |x+y|/|x| they use |x+y|/max(|x|,|y|), because the addition should be commutative. But they don't. So, for the HP 30S, 1e12 + 1 <> 1 + 1e12, and

1e12 + 1 - 1e12 = 0
1 + 1e12 - 1e12 = 1

Horrible! They build a calculator with excellent internal precision and blow it up by a stupid rounding algorithm.