▼
Posts: 305
Threads: 17
Joined: Jun 2007
Suppose we continue to embrace the same philosophy I described in the first post on this topic, "don't commit avoidable error".
My discussion is limited to calculators that do finite precision BCD floating point arithmetic.
In the V8N1 issue of Datafile, Wlodek MierJedrzejowicz had an article about the "Savage" benchmark. The benchmark is called "Savage" not because it is vicious, but because Bill Savage presented it in Byte magazine.
It is, in HP71 Basic:
5 RADIANS
10 A=1
20 FOR I=1 TO 2499
30 A=TAN(ATN(EXP(LOG(SQR(A*A)))))+1
40 NEXT I
50 PRINT A
Upon first examination, one thinks that the calculator should get a result of 2500.00000000 if it did everything as it should. But this is not so. The HP71 actually gets 2499.99948647.
Should we be disappointed by this result? (All the other Saturn based machines, such as the HP48 get this result also.) What result should we expect, if not 2500.00000000?
For the true masochists, what should a calculator get that does n digit arithmetic, 8 <= n <= 16?
▼
Posts: 177
Threads: 12
Joined: Jan 1970
Hi Rodger. This is an interesting exercise. On the 33s I got 2,499.99942403  not exactly 2,500 but very close. Is it close enough? Can someone verify this result?
Thanks,
John
▼
Posts: 305
Threads: 17
Joined: Jun 2007
John,
Did you read the first thread, "What should we get?", (without a part no.)?
The idea here is that if a 12 digit calculator carries out this "benchmark" , it isn't going to get exactly 2500, and you shouldn't expect that it would. And, in fact, you shouldn't give more credit to a (ndigit) calculator that gets closer to 2500 than (another ndigit) one that doesn't. You should give points to the ndigit calculator that get closest to what an ndigit calculator should get, and that is never 2500 exactly.
Of course, the result obtained by a calculator with more digits is going to be closer to 2500 than that obtained by a calculator with fewer digits, but neither will be exactly 2500 if it does proper BCD floating point arithmetic.
The philosophy that HP's floating point arithmetic embodies is that every individual calculation should return what one would get if it were done with infinite precision and then properly rounded to an ndigit result.
And, if this is done, there will be inevitable errors caused by the fact that the result of one calculation is only n digits wide, and then will be used as the input argument of the next calculation.
But, there will be only one "correct" result at every step of the way if the aforementioned philosophy is followed.
So, the way to evaluate a calculator is to first see if it's doing what it should do (for an ndigit calculator), before comparing it to other calculators.
In fact, if all calculators did their arithmetic properly (that is, according to the philosophy above), all you would have to do to know how well a given calculator performs would be to find out how many digits it carried; all ndigit calculators would behave the same if only they all did their arithmetic correctly. I'm referring to those mathematical operations required to be done properly by the IEEE Floating Point standard, such as +, , *, /, SQRT, etc. The performances of higher functions such as trig and other transcendentals are not specified, and may have larger errors. When analyzing the performance of a particular calculator on the Savage benchmark, if the errors involve those unqualified functions, that is more excusable than if an error occurs with SQRT or *.
In the case of this Savage benchmark, 12digit calculators that round to even should get 2499.99942402 as a final result. I am astounded that the 33S does as well as it does, especially in light of the errors in the TAN function recently discussed.
The HP71 performance is almost as good. It makes 3 very tiny mistakes. The first occurs when the loop variable I is 1470. At that point, the arctangent of 1469.99992223 must be calculated. The exact result is 1.5701160547549999736+; the HP71 returns 1.57011605476. It shouldn't have rounded up the trailing ...475 to ...476. But, it would have to have calculated the unrounded result to more than its internal 15 digits to have made the proper decision, so I excuse it!
What does the 33S return for ATN(1469.99992223)?
The HP71 makes similar mistakes when I = 1648 and when I = 1690. The first two recover after more iterations, but the last one doesn't, and leads to a final result of 2499.99948647. If you trap the program when I = 1690 and substitute the correct result for the ATN, then let the program resume, the the final result is 2499.99942402. Knowing what the final result should be for an ndigit calculator allows one to make an initial assessment of a calculator. If the calculator gets what it should, you have evidence that the calculator is doing very good arithmetic. If it doesn't, then you can compare the intermediate results with some known good results (use a PC based mathematics program to get known accurate values), and find out where and how often the calculator makes a mistake. If they aren't the excusable sort the HP71 makes, then you can evaluate the quality of the arithmetic done by the calculator.
In Wlodek's article, he gives a result for the HP41 of 2499.970322; is this what we should expect?
For the HP75C, he gives a result of 2499.99942403, the same result you got for the 33S. However, I think the result should be 2499.99942402. I wonder where that 1 count difference in the LSD is coming from. I might have to break out the HP75C.
No ndigit calculator, 8 <= n <= 16, should get exactly 2500 as a final result from this benchmark. If it does, then it is committing "avoidable error".
The HP71 has the property that its rounding modes can be varied, and you get different results from the Savage benchmark in each case. Mini challenge: Determine what a 12digit calculator should get for the 4 rouding modes available in the HP71. Does the HP71 get those results?
And, as I said in the first post:
"For the true masochists, what should a calculator get that does n digit arithmetic, 8 <= n <= 16?"
Of course, I meant to make the assumption that proper roundtoeven is done, and the philosophy discussed above is followed.
▼
Posts: 177
Threads: 12
Joined: Jan 1970
Quote:
What does the 33S return for ATN(1469.99992223)?
The 33s returns 1.57011605476.
Regards,
John
▼
Posts: 305
Threads: 17
Joined: Jun 2007
So it does what the Saturn machines do.
But, as I mentioned earlier, the calculator can recover from this error. The error made by the Saturn machines, and from which they don't recover is the calculation of ATN(1689.99993538).
What does the 33S get for this calculation?
If it gets 1.57020461086 rather than the 1.57020461087 which the Saturn machines get, then that would explain why it gets almost exactly the proper result for the whole benchmark.
▼
Posts: 177
Threads: 12
Joined: Jan 1970
Quote:
But, as I mentioned earlier, the calculator can recover from this error. The error made by the Saturn machines, and from which they don't recover is the calculation of ATN(1689.99993538).
What does the 33S get for this calculation?
If it gets 1.57020461086 rather than the 1.57020461087 which the Saturn machines get, then that would explain why it gets almost exactly the proper result for the whole benchmark.
Rodger the 33s returns 1.57020461086.
Reards,
John
Posts: 8
Threads: 2
Joined: Jan 1970
I would be willing to bet that the number of devices that can distiguish betwee 2499.99948647 and 2500 (if any exist at all) can be counted on one hand.
In the real world, if you can build something accurate to 3 significant digits, you are doing good. To 4 significant digits, you are doing very good. 5 significant digits is what the highest quality ball bearings are made to. Anything over 5 is amazing.
In other words, 2499.99948647 is plenty close enough to 2500.
However, it is easy to get round off errors that are much greater than that and would be cause for concern. Try the quadradic formula when b is very large and a and c are very small.
Chris W
▼
Posts: 776
Threads: 25
Joined: Jun 2007
"I would be willing to bet that the number of devices that can distiguish betwee 2499.99948647 and 2500 (if any exist at all) can be counted on one hand"
I'll take your bet! There are numerous brands and models (and thousands of units!) of frequency counters that are precise to 9 or 10 digits (and that accurate, too, if they are being fed a proper frequency reference: a cesium beam frequency standard is good to parts in 10E12 absolute, and a hydrogen maser is relatively stable to parts in 10E14 on tens of minutes time scales).
Posts: 177
Threads: 12
Joined: Jan 1970
Rodger, it would be interesting to see how a 30s / 9g would perform with this test. I have a 30s but it's not programmable and I don't relish hitting a combination of keys 2,500 times. Is anyone who owns a 9g interested in trying this?
Regards,
John
▼
Posts: 16
Threads: 0
Joined: Jan 1970
Note that the iterative part of the Savage Benchmark is for testing the system performance of the various functions involved.
The "2500" is simply the exact mathematical result for the calculation performed by the last iteration of the loop (when A=2499). So no programming is required to see how closely your 30S or 9G comes to 2500.
More details on the Savage Benchmark can be found here:
<http://www.technicalc.org/tiplist/en/files/pdf/tips/tip6_50.pdf>
Mark
▼
Posts: 776
Threads: 25
Joined: Jun 2007
That's not right, at least as given by Rodger's BASIC listing.
This line: 30 A=TAN(ATN(EXP(LOG(SQR(A*A)))))+1
makes the value of A depend on all that has happened previously in the loop. If A*A is replaced by I, then your statement is correct.
▼
Posts: 776
Threads: 25
Joined: Jun 2007
That should be I*I , not just I, in my last sentence.
Posts: 177
Threads: 12
Joined: Jan 1970
Quote:
That's not right, at least as given by Rodger's BASIC listing.
This line: 30 A=TAN(ATN(EXP(LOG(SQR(A*A)))))+1
makes the value of A depend on all that has happened previously in the loop. If A*A is replaced by I, then your statement is correct.
I agree Dave, because it is iterative the error accumulates.
For any who is interested, here is the code I wrote for the 33s to solve this:
LBL A
1
STO A
STO I
LBL B
RCL A
x^2
\x (Square root of x)
LN
e^x
ATAN
TAN
1
+
STO A
1
STO+ I
RCL I
2,499
x>=y?
GTO B
RCL A
RTN
Comments are welcome.
Regards,
John
▼
Posts: 177
Threads: 12
Joined: Jan 1970
Out of curiousity I coded this challenge on the TI 83+ SE and the result was 2,499.999992.
Here's my code:
1>A
For(I,1,2499)
tan(tan^{1}(e^(ln(\(A^{2})))))+1>A
End
Disp A
Comments are welcome.
Regards,
John
▼
Posts: 2,761
Threads: 100
Joined: Jul 2005
From the aforementioned Wlodek MierJedrzejowicz's article:
"...on older HP calculators trig is accurate to almost 12 digits, but the policy of removing the last two digits, instead of leaving them hidden, can lead to less accuracy too. Newer HP calculators with a Saturn CPU work to 15 digits precision (and accuracy) internally, but round results to only 12 digits at the end of each
calculation  the TI74 works to 14 digits accuracy and precision, and leaves all 14 digits on the stack (with 4 hidden digits), so the final accuracy of the TI74 result is 2500.0000291436."
Truncating the extra internal digits as HP does increases cumulative errors needlessly. Consider, for instance,
Free42 which returns 2500.000000000000729718363 in this benchmark. Had it followed HP's philosophy the result would have been 2499.99948647, which is much less accurate. This feature could be left as an option to the user, though.
Also out of curiosity, I ran the benchmark on my first (acutally on an emulator: http://codigolivre.org.br/frs/?group_id=1367&release_id=1598) and my second computer. The results were 2500.00009 and 2500.0002592983, respectively.
By the way, the original benchmark program can be found here (Page 12, Column 1) :
http://www.amigau.com/68K/dg/dg25.htm
Regards,
Gerson.
▼
Posts: 305
Threads: 17
Joined: Jun 2007
Quote: Truncating the extra internal digits as HP does increases cumulative errors needlessly. Consider, for instance, Free42 which returns 2500.000000000000729718363 in this benchmark. Had it followed HP's philosophy the result would have been 2499.99948647, which is much less accurate. This feature could be left as an option to the user, though.
HP doesn't truncate the internal digits; it rounds to even.
HP's philosophy is to perform each calculation as if it were done with infinite precision (where possible, and it is possible with the 4 basic arithmetic operations and square root, etc.), and then round (to even) the result to the (maximum) n digit number displayed by the calculator.
This is the best result that can be had with an n digit calculator.
There seem to be 3 basic techniques to be found in calculators.
1. Do the calculations with n+k digits, keep n+k digits for subsequent calculations, but only display n digits. This is the TI way, with n=12 and k=2 for the TI86, for example.
2. Do the calculations with n+k digits, but round the result to n digits for display and subsequent calculations. This is the HP way, with n=12 and k=3 in Saturn machines.
3. Do everything with n digits, and don't bother with extra digits to get properly rounded results. Many lowend calculators seem to use this method.
I don't know that I would say that method 2 "...increases cumulative errors needlessly". Any method accumulates error, simply because finite precision n digit BCD floating point arithmetic is only finite precision.
The merits and demerits of the first two methods have been argued to death over the years, and I'm not going to get into it again here.
2500.000000000000729718363 is a 25 digit result. Apparently Free42 does at least 25 digit calculations, but I don't know if it uses extra internal digits or not (or how many). How could following HP's philosopy (which isn't to truncate the extra internal digits, if there are any), reduce a 25 digit result to a 12 digit result? The HP philosophy is to round the internally calculated result to n digits for an n digit calculator, which is apparently 25 digits on Free42.
But, if Free42 were to round each calculation to 12 digits (the better to emulate a real HP42) before displaying and before using the result in a subsequent calculation, (in effect an HP42 which had 13 extra internal digits), then it would give a result as though the emulated HP42 didn't make the mistake I mentioned in an earlier post:
"The HP71 performance is almost as good. It makes 3 very tiny mistakes. The first occurs when the loop variable I is 1470. At that point, the arctangent of 1469.99992223 must be calculated. The exact result is 1.5701160547549999736+; the HP71 returns 1.57011605476. It shouldn't have rounded up the trailing ...475 to ...476."
If Free42 did this, then it would get 2499.99942402 rather than 2499.99948647, because the extra 13 internal digits would be more than sufficient to avoid the tiny mistakes that lead to a result of 2499.99948647 on real Saturn based machines.
This demonstrates one way in which simulators/emulators may not get the same result as the actual calculator they are emulating. The HP42 is a 12 digit machine, but Free42 carries more digits than that, and therefore returns a more precise (and probably more accurate) result than a real HP42.
The old HP71 Forth emulator for the HP41 had the same characteristic. Since the HP71 used 12 digits, you would often get a different result from a calculation than a real HP41, with its 10 digits.
If Free42 carries only 25 digits for the user accessible results (has no hidden digits as TI calculators are wont to have), it should have gotten 2500.00000 00000 00000 01104 9 rather than 2500.00000 00000 00729 71836 3, had it followed HP's philosophy (round to even).
If Free42 performed each separate calculation as though with infinite precision, but then applied different rounding rules, the results should have been:
Rounding mode Final result
Round up 2500.00000 00000 00002 59023 5
Truncate 2499.99999 99999 99997 46933 3
It appears that Free42's actual result, 2500.00000 00000 00729 71836 3, is worse than any of these, so I wonder what it's doing for arithmetic? At best, it's doing what the HP30S does. Rather than do proper arithmetic, just overwhelm the problem by carrying many extra digits, and throw a lot of them away in the final results.
This is easier to do nowadays given the state of the art in microelectronics. But in the early days of calculators, carrying 25 digits to get a final result of 12 digits was a (not very elegant) shortcut the designers of the day couldn't afford.
▼
Posts: 2,761
Threads: 100
Joined: Jul 2005
Thanks for your convincing explanation and sorry for my confusion between truncation and rounding. I was fixing a program this afternoon because of a truncation problem and had kept the word in my mind.
Being a layman in this matter I may have been impressed by the nice 2500 showing in the Free42 displayed after twentyfive hundred iterations, the same way people got impressed with the HP30S returning a perfect 9 in the forensic test.
Posts: 3,283
Threads: 104
Joined: Jul 2005
Here is the program for the 9G:
INPUT N;
A=1;
FOR(I=1;I<N;++I){
A=tan(tan^{1}(e^(ln(/(A*A)))))+1;
}
PRINT A;
END;
/ denotes the square root operator. ++ is a special operator found in the INST menu.
The program editor is horrible, since you can easily (and accidently) split lines but I didn't find a way to join the fragments again!
The result is interesting: exactly 2500. AN returns zero.
Marcus
Edited: 7 Dec 2006, 5:06 a.m.
▼
Posts: 305
Threads: 17
Joined: Jun 2007
Have you tried to determine the characteristics of the 9G?
How many digits does it carry, and are any of them hidden?
Is it BCD or binary internally?
▼
Posts: 3,283
Threads: 104
Joined: Jul 2005
The 9g seems to work in binary:
1/3 returns 0.333333333 to the display.
Ans*1E9INT(Ans*1E9) returns the following sequence if applied repeatedly:
0.333333333
0.333333037
You can press Enter several more times before you get a zero in the display.
So it has 22 significant digits but returns garbage thereafter.
Marcus
▼
Posts: 305
Threads: 17
Joined: Jun 2007
It would be interesting if you would apply some of the tests I applied to the HP30S in this thread:
http://www.hpmuseum.org/cgisys/cgiwrap/hpmuseum/archv015.cgi?read=85973#85973
▼
Posts: 3,283
Threads: 104
Joined: Jul 2005
I'm too lazy to test it all, but a quick check for the standard forensics expression returns an exact 9. So I assume, the 9G is using the same algorithms as the 30s.
Marcus
▼
Posts: 305
Threads: 17
Joined: Jun 2007
Could you try the last test in that thread:
"So finally, this is why the HP30S gets exactly 9.00000000000000000 when running the calculator forensics test. To see the real error, one should subtract the input argument of the test from the final result. That is, do:
arcsin(arcos(arctan(tan(cos(sin(9))))))9
On the HP30S, the result is 0, leading one to believe that the HP30S is *really* accurate. But redo the test as:
arcsin(arcos(arctan(tan(cos(sin(9.1))))))9.1
to foil its rounding near integers and see a result: 4.833288903E11"
If you get the same error as the HP30S for the forensics test with a starting value of 9.1 instead of 9, then I would consider that good evidence that the 9G is using the same algorithms, and probably even the same firmware engine.
▼
Posts: 3,283
Threads: 104
Joined: Jul 2005
Quote: arcsin(arcos(arctan(tan(cos(sin(9.1))))))9.1
I get 1.056805951 x 10^{18}
So the results are, in fact, different.
Posts: 177
Threads: 12
Joined: Jan 1970
Quote:
The result is interesting: exactly 2500. AN returns zero.
Thanks Marcus. What about (AN)*1e6?
Regards,
John
▼
Posts: 3,283
Threads: 104
Joined: Jul 2005
Quote: What about (AN)*1e6?
If the result of AN were nonzero, the calc should have indicated it, shouldn't it? The result looks very much like an exact integer. This is probably a "special rounding" because all input is integer, too.
Marcus
