calc precision « Next Oldest | Next Newest »

 ▼ mapet Member Posts: 57 Threads: 27 Joined: Jan 1970 02-10-2003, 09:51 AM Which of the HP calculators uses the highest precision? ▼ Ex-PPC member Member Posts: 142 Threads: 24 Joined: Jan 1970 02-10-2003, 10:03 AM HP30S (24 digits), except for square root (12 digits) ▼ Bill Wiese Senior Member Posts: 308 Threads: 26 Joined: Jul 2007 02-10-2003, 02:15 PM >>HP30S (24 digits), except for square >>root (12 digits) Wow. Why is sqrt(x) lagging? Common square root algorithms work very similarly to division & reciprocal algorithms. Unless they totally ran out of ROM space and picked a reduction in sqrt(X) precision as the best compromise, this looks like an oversight. I believe - it's been awhile - that there's a CORDIC-like shift/add algorithm for square root too - each iteration would deliver one digit (either binary bit, hex digit or BCD digit, according to base of numeration system and algorithm flavora) of result. Square root is fairly well-behaved too - unlike asymptotic tan(x) near 90deg and log(x), ln(x) near 0. Bill Wiese San Jose, CA mapet Member Posts: 57 Threads: 27 Joined: Jan 1970 02-11-2003, 07:58 AM nt hugh Member Posts: 189 Threads: 23 Joined: Jan 1970 02-13-2003, 07:50 PM i have also noticed square root is less accurate. if you root with logs you can get more accuracy that the built in square root. that said, its not necessarily a recommendation for the hp30s.

 Possibly Related Threads... Thread Author Replies Views Last Post HP Prime numerical precision in CAS and HOME Javier Goizueta 5 591 11-16-2013, 03:51 AM Last Post: Paul Dale wp34s calc to calc communication Andrew Nikitin 1 338 07-29-2013, 01:57 AM Last Post: Marcus von Cube, Germany [wp34s] Converting to/from IEEE 754 Binary64 (Double Precision) David Maier 1 262 06-15-2013, 09:21 PM Last Post: Paul Dale Precision DC-50 Mic 3 431 05-26-2013, 01:27 PM Last Post: DavidM Estimating accuracy in finite precision computations mpi 17 1,131 02-22-2013, 09:44 AM Last Post: mpi 50G precision & accuracy Matt Agajanian 11 766 05-17-2012, 11:15 AM Last Post: Crawl WP 34S: Double precision registers - What are your feelings? Marcus von Cube, Germany 16 1,039 12-26-2011, 10:49 AM Last Post: Dieter [WP34s] Questions about extended precision fhub 7 529 10-28-2011, 05:20 PM Last Post: Marcus von Cube, Germany HP 35s. FIX affects precision Pablo P (Spain) 4 373 10-22-2010, 11:48 PM Last Post: Karl Schneider High precision calculator in Windows 7 jotne 41 2,060 08-21-2009, 09:19 PM Last Post: Eric Smith

Forum Jump: