HP Forums
calc precision - Printable Version

+- HP Forums (https://archived.hpcalc.org/museumforum)
+-- Forum: HP Museum Forums (https://archived.hpcalc.org/museumforum/forum-1.html)
+--- Forum: Old HP Forum Archives (https://archived.hpcalc.org/museumforum/forum-2.html)
+--- Thread: calc precision (/thread-27683.html)

calc precision - mapet - 02-10-2003

Which of the HP calculators uses the highest precision?

Re: calc precision - Ex-PPC member - 02-10-2003

HP30S (24 digits), except for square root (12 digits)

Re: calc precision - Bill Wiese - 02-10-2003

>>HP30S (24 digits), except for square
>>root (12 digits)

Wow. Why is sqrt(x) lagging? Common square root algorithms work very similarly to division & reciprocal algorithms. Unless they totally ran out of ROM space and picked a reduction in sqrt(X) precision as the best compromise, this looks like an oversight.

I believe - it's been awhile - that there's a CORDIC-like shift/add algorithm for square root too - each iteration would deliver one digit (either binary bit, hex digit or BCD digit, according to base of numeration system and algorithm flavora) of result. Square root is fairly well-behaved too - unlike asymptotic tan(x) near 90deg and log(x), ln(x) near 0.

Bill Wiese
San Jose, CA

HP 9G? (nt) - mapet - 02-11-2003


Re: calc precision - hugh - 02-13-2003

i have also noticed square root is less accurate. if you root with logs you can get more accuracy that the built in square root.
that said, its not necessarily a recommendation for the hp30s.