A few years ago, my boss suggested the HP48sx was not an accurate instrument based on his misunderstanding the following article:

Article here.

The article defines two equations for calculating great circle distances and states the first equation listed is not very accurate depending on what calculation device or software is used. The author states the second equation is more accurate for all devices.

My boss assumed the HP48sx was not accurate due to the error margin in the result of the first equation. I would like to argue any TI calc would give similar results to the 48sx when using equation 1.

In a nutshell:

Equation 1 evaluated with Excel is accurate.

Equation 1 evaluated with HP48sx is not very accurate.

Equation 1 evaluated with TI-XX might not be very accurate either.

Equation 2 evalutated with Excel is accurate.

Equation 2 evaluated with HP48sx is accurate.

Equation 2 evaluated with TI-XX should be accurate.

Would equation 1 evaluated with a TI-XX be more accurate than an HP48SX? Unfortunately, the author of the article did not include several variables, such as a pair of lat/lons, or earth radius.

edit: fixed link.

Edited: 9 Sept 2009, 7:11 p.m.

It would be interesting for you to share the actual data that you had used in your testing.

TomC

The first formula (based on the spherical law of cosines) is ill-conditioned for small distances and will have accuracy problems on almost any computer; it is simply much more sensitive to limited precision than the haversine formula. This is an example of why numerical analysis is important. Just because you have a mathematically correct formula doesn't mean that it will be generally useful on a computer (or calculator).

See also "Virtues of the Haversine", R.W. Sinnott, Sky and Telescope, vol. 68, no. 2, 1984, p. 159.

the idea of a great circle is somewhat misleading in the context of earth navigation. use the spherical formulae for pure spheres, but for earth distances it's a bad idea.

for some time i've been using Meeus' formula. i put some code here, http://www.voidware.com/earthdist.htm

but recently, i became aware of a formula by Vincenty, http://en.wikipedia.org/wiki/Vincenty's_formulae. I read the paper and coded it up. it's a bit more complicated, but claims to be super accurate. However, i had problems with it not converging for points almost opposite the globe.

in any case, unless you really want to measure distances on WSG84, you're going to have a height difference involved which the, as presented, Vincenty formula doesn't deal with. so, im still happy to use the Meeus approximation for real use.

However, at some point im going to have to try to get something that takes height into account too.