Precision of HP 30S



I read that the 30S works with binary representations of numbers, whereas most other pocket calculators use decimal numbers. And I read that its precision is equivalent to 24 decimal places.

Does anybody have an explanation for following results?

(a) 1E10 + 0.11 - 1E10 gives 0.11
(b) 1E10 + 0.01 - 1E10 gives 0

Instead of 0.11 in (a) you can use 0.12345678 also and still get the correct result. So what's happening in (b)???

Things like these make it rather difficult to rely on the precision of this calculator or to estimate error bounds.



the hp30s and 9g models do, in fact, work in binary. they also, generally get the correct answers. but there are drawbacks to binary requiring the extra internal precision and also some devious value suppression going on.

its the supression that's expressing the behaviour you've seen. i wrote some stuff about working in binary here:


I think a rounding scheme like that makes the 24 internal digits somewhat useless. I don't see a rational justification for supressing x=0.01, x=0.02, and x=0.03 when calculating 1E10+x but keeping all digits of x=0.11 or even of x=0.12345678.

Are you able to explain the mechanism a bit more detailed? It seems that 1E10+0.01 is changed to 1E10, but 1E10+0.11 is kept. What's the exact rule? Suppose the result of an operation is a. What does the calculator do with this number? Your rounding algorithm needs a second number b to check whether a~b. How does it find b? It seems to me that the decimal expansion of the numbers is used here. Doesn't this imply that a decimal number representation is used?



its because 0.11 is roughly ten times bigger than 0.01.

here’s how i think it works (really i’m guessing). it doesn’t just use the final answer `a’. it compares `a’ to one of the terms of the input ‘b’. this suppression only applies to ADD and SUB at the top level and not to adds and subs used internally.

define ADD(x, y) to be 0 if |(x + y)/x| < eps, where eps = 1e-12, say.
and ADD(x,y) to be x + y, otherwise.

define SUB(x, y) to be 0 if |(x - y)/x| < eps, where eps = 1e-12, say.
and SUB(x,y) to be x - y, otherwise.

in the above “+” means the result of the floating point binary add (ie not true real number addition).

so, in your examples. ex1: 1e10 + 0.11 - 1e10 = 0.11
we have SUB(ADD(1e10,0.11), 1e10) = SUB(10000000000.11,1e10) = 0.11 because
(10000000000.11-1e10)/10000000000.11 = 1.0999999..e-11 > eps

but, ex2: 1e10 + 0.01 - 1e10 = 0 because
SUB(ADD(1e10,0.01),1e10) = SUB(10000000000.01,1e10) = 0, since
(10000000000.01-1e10)/10000000000.01 = 9.9999999..e-13 < eps


Your algorithm seems to be correct, thanks. Maybe eps=2e-12 is actually used.

My first thought was that certainly instead of |x+y|/|x| they use |x+y|/max(|x|,|y|), because the addition should be commutative. But they don't. So, for the HP 30S, 1e12 + 1 <> 1 + 1e12, and

1e12 + 1 - 1e12 = 0
1 + 1e12 - 1e12 = 1

Horrible! They build a calculator with excellent internal precision and blow it up by a stupid rounding algorithm.

Possibly Related Threads…
Thread Author Replies Views Last Post
  HP Prime numerical precision in CAS and HOME Javier Goizueta 5 2,421 11-16-2013, 03:51 AM
Last Post: Paul Dale
  [wp34s] Converting to/from IEEE 754 Binary64 (Double Precision) David Maier 1 1,131 06-15-2013, 09:21 PM
Last Post: Paul Dale
  Precision DC-50 Mic 3 1,534 05-26-2013, 01:27 PM
Last Post: DavidM
  Estimating accuracy in finite precision computations mpi 17 4,367 02-22-2013, 09:44 AM
Last Post: mpi
  50G precision & accuracy Matt Agajanian 11 3,049 05-17-2012, 11:15 AM
Last Post: Crawl
  WP 34S: Double precision registers - What are your feelings? Marcus von Cube, Germany 16 4,396 12-26-2011, 10:49 AM
Last Post: Dieter
  [WP34s] Questions about extended precision fhub 7 2,206 10-28-2011, 05:20 PM
Last Post: Marcus von Cube, Germany
  HP 35s. FIX affects precision Pablo P (Spain) 4 1,493 10-22-2010, 11:48 PM
Last Post: Karl Schneider
  High precision calculator in Windows 7 jotne 41 9,253 08-21-2009, 09:19 PM
Last Post: Eric Smith
  HP41 Mcode Entrypoint for 'high precision numbers' PeterP 3 1,204 07-30-2008, 02:37 PM
Last Post: Eric Smith

Forum Jump: