HP48SX accuracy questioned.


A few years ago, my boss suggested the HP48sx was not an accurate instrument based on his misunderstanding the following article:

Article here.

The article defines two equations for calculating great circle distances and states the first equation listed is not very accurate depending on what calculation device or software is used. The author states the second equation is more accurate for all devices.

My boss assumed the HP48sx was not accurate due to the error margin in the result of the first equation. I would like to argue any TI calc would give similar results to the 48sx when using equation 1.

In a nutshell:

Equation 1 evaluated with Excel is accurate.

Equation 1 evaluated with HP48sx is not very accurate.

Equation 1 evaluated with TI-XX might not be very accurate either.

Equation 2 evalutated with Excel is accurate.

Equation 2 evaluated with HP48sx is accurate.

Equation 2 evaluated with TI-XX should be accurate.

Would equation 1 evaluated with a TI-XX be more accurate than an HP48SX? Unfortunately, the author of the article did not include several variables, such as a pair of lat/lons, or earth radius.

edit: fixed link.

Edited: 9 Sept 2009, 7:11 p.m.


It would be interesting for you to share the actual data that you had used in your testing.



The first formula (based on the spherical law of cosines) is ill-conditioned for small distances and will have accuracy problems on almost any computer; it is simply much more sensitive to limited precision than the haversine formula. This is an example of why numerical analysis is important. Just because you have a mathematically correct formula doesn't mean that it will be generally useful on a computer (or calculator).

See also "Virtues of the Haversine", R.W. Sinnott, Sky and Telescope, vol. 68, no. 2, 1984, p. 159.


the idea of a great circle is somewhat misleading in the context of earth navigation. use the spherical formulae for pure spheres, but for earth distances it's a bad idea.

for some time i've been using Meeus' formula. i put some code here, http://www.voidware.com/earthdist.htm

but recently, i became aware of a formula by Vincenty, http://en.wikipedia.org/wiki/Vincenty's_formulae. I read the paper and coded it up. it's a bit more complicated, but claims to be super accurate. However, i had problems with it not converging for points almost opposite the globe.

in any case, unless you really want to measure distances on WSG84, you're going to have a height difference involved which the, as presented, Vincenty formula doesn't deal with. so, im still happy to use the Meeus approximation for real use.

However, at some point im going to have to try to get something that takes height into account too.

Possibly Related Threads...
Thread Author Replies Views Last Post
  How much accuracy does one actually need? Matt Agajanian 23 2,269 08-26-2013, 12:46 PM
Last Post: Kimberly Thompson
  Estimating accuracy in finite precision computations mpi 17 1,811 02-22-2013, 09:44 AM
Last Post: mpi
  HP41 emulator for HP48sx - keyboard overlay needed Doug (NYC) 3 772 12-31-2012, 03:11 PM
Last Post: Allen
  HP48SX seems fully functional won't run SELF TESTS? Bruce Larrabee 5 919 07-16-2012, 11:04 PM
Last Post: Luiz C. Vieira (Brazil)
  WP 34S accuracy is excellent Jeff Johnson 15 1,556 06-01-2012, 10:41 PM
Last Post: Valentin Albillo
  50G precision & accuracy Matt Agajanian 11 1,333 05-17-2012, 11:15 AM
Last Post: Crawl
  Accuracy of Woodstocks Matt Agajanian 7 930 03-25-2012, 06:54 PM
Last Post: Eric Smith
  HP-35 Accuracy Threshhold Matt Agajanian 0 335 03-22-2012, 09:56 PM
Last Post: Matt Agajanian
  accuracy of integration and solve routines HP 15C LE Jan 3 595 02-02-2012, 01:03 PM
Last Post: Marcus von Cube, Germany
  Did We Miss The 20th Anniv. of the HP48SX? Jake Schwartz 6 796 03-19-2010, 05:55 AM
Last Post: Bill Zimmerly

Forum Jump: