calc precision - Printable Version +- HP Forums ( https://archived.hpcalc.org/museumforum)+-- Forum: HP Museum Forums ( https://archived.hpcalc.org/museumforum/forum-1.html)+--- Forum: Old HP Forum Archives ( https://archived.hpcalc.org/museumforum/forum-2.html)+--- Thread: calc precision ( /thread-27683.html) |

calc precision - mapet - 02-10-2003
Which of the HP calculators uses the highest precision?
Re: calc precision - Ex-PPC member - 02-10-2003
HP30S (24 digits), except for square root (12 digits)
Re: calc precision - Bill Wiese - 02-10-2003
>>HP30S (24 digits), except for square Wow. Why is sqrt(x) lagging? Common square root algorithms work very similarly to division & reciprocal algorithms. Unless they totally ran out of ROM space and picked a reduction in sqrt(X) precision as the best compromise, this looks like an oversight. I believe - it's been awhile - that there's a CORDIC-like shift/add algorithm for square root too - each iteration would deliver one digit (either binary bit, hex digit or BCD digit, according to base of numeration system and algorithm flavora) of result. Square root is fairly well-behaved too - unlike asymptotic tan(x) near 90deg and log(x), ln(x) near 0.
Bill Wiese HP 9G? (nt) - mapet - 02-11-2003
nt
Re: calc precision - hugh - 02-13-2003
i have also noticed square root is less accurate. if you root with logs you can get more accuracy that the built in square root. |