More on testing calculators and computers



#2

In a recent thread Rodger Rosenbaum suggested:

Quote:
Try doing the sin cos tan atan acos asin test all on one command line with starting values of 6,7,8,10,11, and 12 degrees and see if the results are all as good as when starting with 9 degrees.

I responded with the results with my Durabrand 828. I have since completed tests over the same input value range with several other machines as shown in the following table:

   N                                                    N - f(N)

Durabrand TI-30 HP-41/67 TI-59 HP-28 TI-85

5 +3.15E-09 -1.67E-01 +1.20E-04 -3.60E-05 +3.35E-06 -7.27E-08
6 +3.23E-09 -1.69E-01 -4.44E-04 +4.32E-08 +1.64E-06 +3.36E-08
7 +2.73E-09 -1.07E-01 +4.80E-04 -6.55E-05 -3.18E-06 +6.49E-08
8 +6.02E-10 -1.68E-01 -2.88E-04 -1.60E-05 +2.88E-06 +9.61E-09
9 +1.92E-09 -1.77E-01 -4.17E-04 -4.66E-06 +1.36E-06 +3.04E-08
10 +5.93E-10 -1.53E-01 +1.83E-04 -5.28E-05 +3.35E-06 +2.99E-08
11 +3.79E-10 -4.49E-02 -3.40E-04 -5.21E-05 +2.47E-06 +2.20E-08
12 +4.07E-10 -8.59E-02 -1.09E-04 -2.91E-05 +1.55E-06 -3.62E-08
13 +6.10E-10 -1.05E-01 -3.07E-05 -4.12E-05 +1.89E-07 -1.67E-08
14 +8.09E-09 -9.36E-02 -5.02E-05 -3.12E-05 -5.47E-07 -1.81E-08
15 +1.43E-09 -6.49E-02 -1.27E-04 -3.12E-05 -2.97E-07 -1.15E-08
20 +8.93E-10 -5.29E-02 -1.28E-04 -7.38E-06 -8.10E-07 -9.06E-09
25 +5.44E-10 -2.20E-02 +2.74E-05 -9.42E-06 -5.76E-07 -6.33E-09
30 +3.09E-10 -4.53E-02 +5.90E-05 -1.78E-06 -9.43E-07 +3.78E-09

where getting the results in tabular form takes more time than getting them. As expected the machines which carried more digits yielded better results. I note that only the TI-59 showed one result (for 6) which was much better than the typical values.

In the same thread Rodger proposed some arithmetic tests of calculators and computers which do not use higher math functions:

Quote:
Here is a test that only checks the caalculator's basic arithmetic accuracy. It is based on the observation that if you reciprocate certain integers twice, you don't get the original integer back, and indeed you shouldn't on a finite precision BCD machine.

For example, type 6 1/x 1/x and you should see 5.99999999... on any calculator that does rounded arithmetic properly. If you see exactly 6 then the calculator isn't displaying all the digits in the result, or it's a "pleaser", like the HP30 (you should get 6.00000...0002 if it truncates properly).

So, the idea is to program a loop, apply 1/x twice for a range of integers, each time subtracting the result from the original integer that was reciprocated, and summing the absolute values of those small differences. This is another of my "what should we get" tests, because the result is determinate; there is a correct result, depending on how many digits are used, and the rounding mode.

Here's the program:


10 N=500 REM NUMBER OF ITERATIONS

20 S=0 REM INITIALIZE RUNNING SUM

30 FOR I = 1 TO N

40 S=S+ABS(I-1/(1/I))

50 NEXT I

60 PRINT S


Rodger gave a result for the HP-71 and suggested "I'm sure you'll want to try it on your own calculators." I started with my old Radio Shack Model 100 which carries thirteen or fourteen digits depending on whether the value to the left of the decimal point has an odd or even number of digits. The sum was 1.9564E-09 .

Rodger also suggested that

Quote:
The square root function is also specified by the IEEE, so line 40 in the program can be modified to:

40 S=S+ABS(I-SQRT(I)*SQRT(I))

and the program should give a specific result on a compliant machine.


I tried that on my Model 100. The sum was 7.9719E-09 . Then, I remembered that there are 22 perfect squares between 1 and 500 which should yield zeros for the value (I-SQRT(I)*SQRT(I)) . I added a couple of lines to the program which would count the occurrences of zeroes. The program reported 84 zeroes instead of the expected 22. I returned line 40 to the original format which tests reciprocals and found that the program reported 300 zeroes.

I tried the reciprocal and square root tests on the HP-41, HP-33s, TI-59, TI-85 and Durabrand 828 with the following results:

                      Reciprocal Test                     Square root - square Test   

Sum Zeroes Sum Zeroes

Model 100 1.956E-09 300 7.972E-09 84

HP-41 6.134E-06 397 3.217E-05 206

HP-33s 6.803E-08 389 3.127E-07 204

TI-59 6.894E-09 402 3.843E-07 22

TI-85 7.353E-10 386 5.743E-09 142

828 0 500 0 500

I don't understand why the TI-59 only gets the expected 22 zeroes. I will try to look harder at how I am doing the Durabrand 828 tests. I remember that back when I was doing square root of two - squared tests on the Casio fx-7000G I had to be careful or adjacent square roots and squares would somehow just cancel each other out.


#3

My HP 20S gives:
5.99999999999 for 6
7.00000000001 for 7
9.00000000001 for 9
etc

Dave.

#4

You report in the text that the Model 100 got 7.9719E-09, but in the table you show 7.972E-09. It would be good to report all the digits you get in the test when you do the table.

The results you report for the 1/x test for the HP-41, HP-33S and TI-85 are exactly what should be gotten for 10, 12 and 14 digit BCD calculators.

The results for the SQRT test for 10, 12 and 14 digit machines should be 3.2127E-05, 3.1267E-07 and 3.0483E-09.

I wonder if the discrepancy for the HP-41 SQRT test is simply because you didn't report all the digits.

I have a TI-86, which I would expect to get the same result as a TI-85. I get the same result you do for the 1/x test, but for the SQRT test I get 3.0483E-09 which is the "what should you get" exact expected result. I was pleasantly surprised to find that the TI-86 did so well on this test and the 1/x test. Evidently TI has cleaned up their basic arithmetic.

Perhaps you could double check your result on the TI-85 SQRT test. Here are some of the results I get for different loop repetition counts.

    N         Result

10 3E-13
20 3E-13
50 4.3E-12
100 2.83E-11

I don't have an HP-33S, but I ran the SQRT test on the HP-71, a 12 digit Saturn machine, and I got 3.1267E-07.

Here are some results from a 12 digit Saturn machine:

    N         Result

10 7E-11
20 7E-11
50 4.7E-10
100 2.67E-9

Perhaps you could also double check your HP-33S results for the SQRT test.

The Model 100 you refer to is the old Radio Shack machine, isn't it? That machine is running Microsoft Basic isn't it? That would mean that it's doing binary arithmetic. I haven't worked out "what we should get" for binary machines, so I can't comment on your results for the 100. :-{

The result for the TI-59 isn't what should be expected for a 13 digit machine. A 13 digit BCD calculator should get 6.204E-07 for the 1/x test, and 3.1777E-08 for the SQRT test. I assume this discrepancy is because the TI-59 didn't do basic arithmetic as well as the later TI calculators.

When I get a bad result in one of these tests that involve a lot of iterations, such as the Savage benchmark with its 2500 iterations, I always wonder where the mistake was made. What I do is to change the number of iterations by half and compare the result with the "correct" result. If there's still an error, then cut the number of iterations in half again. If there's no error this time, then increase the number of iterations to a point halfway to where the last error was made, and so forth until you get to the particular calculation that made the error. This binary search technique is the fastest to zero in on where the error was made. When you get there, then you can examine the individual calculation that made the error in detail and find out what went wrong.


#5

Rodger:

You wrote:

Quote:
I have a TI-86, which I would expect to get the same result as a TI-85. I get the same result you do for the 1/x test, but for the SQRT test I get 3.0483E-09 which is the "what should you get" exact expected result. I was pleasantly surprised to find that the TI-86 did so well on this test and the 1/x test. Evidently TI has cleaned up their basic arithmetic.

Perhaps you could double check your result on the TI-85 SQRT test. Here are some of the results I get for different loop repetition counts.

    N         Result
10 3E-13
20 3E-13
50 4.3E-12
100 2.83E-11


I ran the square root - squared test problem from the keyboard of my TI-85 for integers 1 through 20 with the following results:
 1     0                       11    0
2 0 12 0
3 -1E-13 13 0
4 0 14 0
5 0 15 -1E-12
6 -1E-13 16 0
7 +5E-13 17 0
8 +5E-13 18 0
9 0 19 1E-12
10 0 20 0
I ran my program which sums absolute values from 1 to 10 and got a sum of 1.2E-12 with six zeroes, and from 1 to 20 and got a sum of 3.2E-12 with 14 zeroes. Those values agree with the data from the keyboard. I conclude that my program is OK and that the previously reported result of 5.7932E-09 for 1 through 500 is probably correct. Thus, the TI-85 does not do as well as the TI-86. This may be associated with the release of the TI-85 in about 1992 and the TI-86 in about 1997. There were ten releases of the ROM for the TI-85. The machine I am using has version 10.0 .
.


#6

Hello, I don't think the TI85 went through all versions 1.0 to 10.0, there were (I believe) only 2 or 3 versions sold to the public.

I just got a NOS TI86, do you know how to display the ROM version ?

I feel the TI86 is the equivalent of the HP42S for TI : the last wonderful example of engineer craftmanship. Newer machines, however powerful, had less soul.


#7

You wrote:

Quote:
Hello, I don't think the TI85 went through all versions 1.0 to 10.0, there were (I believe) only 2 or 3 versions sold to the public.

I just got a NOS TI86, do you know how to display the ROM version ?

I feel the TI86 is the equivalent of the HP42S for TI : the last wonderful example of engineer craftmanship. Newer machines, however powerful, had less soul.

I found the information on the ROM versions of the TI-85 at //www.ticalc.org/basics/calculators/ti-85.html where it says that to check the ROM version you press 2nd - MODE - ALPHA - S . I did that and got 10.0. The unit I am using is S/N 48005260. I have an earlier serial number unit in Florida. I will check its version next week.

To check on the ROM versions of the TI-86 go to http://www.ticalc.org/basics/calculators/ti-86.html . It shows that there were versions 1.2 through 1.6 . You can check your version with the same sequence as with the TI-85. Before you try this be sure to read the instructions about NOT pressing ENTER in that mode or you will clear memory.


#8

Thank you, I have version 1.6. That machine was purchased new last week ! Serial number is 14226797 I-06981. The hard clear plastic package contained batteries valid until 2005...

Sorry for off-topicness.

#9

Palmer,

I checked the case of n=8 on the TI-86 (version 1.6) in more detail.

When I do SQRT(8)*SQRT(8), I get 8.0000000000001. When I do SQRT(8)^2, I get exactly 8.

What do you get on your TI-85?

#10

You can see what I was getting at when I asked you to do the sin cos tan atan acos asin test with several input arguments. Most of the calculators' results varied over an order of magnitude. So if you were comparing two calculators, one of them might, by accident, happen to give a especially good result, while the other might happen to give an especially bad result.

And, occasionally, as you found with the TI-59, you might get an *really* good result.

These various tests that only use one value are not particularly good for comparing calculators for this reason.


#11

Roger:

Thank you for your review of the data I presented.

The difference between the value in the text and in the table is simply due to the fact that I rounded all of the answers back to four significant digits in the table. I have the full values in my notes.

I did manage to do the two tests on a TI-55. The sum for the 1/x test was 3.362E-07. The sum for the square root test was 3.494E-06. I was not able to find the number of zeroes because the TI-55 does not provide any comparison tests and I am just not up to checking the answers for the five hundred input values one by one.

I also did the two tests on my HP-12C and got exactly the same results as with the HP-41.

The only other machine that I have here in North Carolina is an HP28S. I just don't do very well programming in RPL.

I am not sure about the Model 100. My recollection is that I did the Paranoia test with it back in the eighties and it came up as base 10 with guard digits and rounding. My recollection is that the Radio Shack Color Computer was base 2. I will try to verify those recollections.

I agree with your comment about the questionable value of doing tests which use only one value.

Palmer


#12

Rodger:

You wrote:

Quote:
The results for the SQRT test for 10, 12 and 14 digit machines should be 3.2127E-05, 3.1267E-07 and 3.0483E-09.

I wonder if the discrepancy for the HP-41 SQRT test is simply because you didn't report all the digits.


You are correct. My exact result for both the HP-41 and the HP-12C as 3.2127E-05. There were 206 zeroes ot of the 500 calculations.

When we were doing comparative tests on machines back in the 1980's it was said that some machines (I think it was the HP's, but I'm not sure, and the results which follow suggest that the supposition was incorrect) had "smarter" y^x algorithms in the sense that if the exponent was an integer, then the machine did repeated multiplications rather than, it was said, taking the logarithm of y, multiplying by x and taking the antilog. The result quoted above was for a program with either the square root multiplied by itself or for the square root squared with the x^2 function. When I did the problem with the square root raised to the second power using the y^x function the result was 3.2137E-05 with 205 zeroes. I didn't mistype. The thousandths digit is really a three.

I wrote a program which compared the results for each integer calculated using the two methods. When the integer was 39 the result with the x^2 method was zero but was 1E-8 with the y^x method. The results were identical for the other 499 integer values. I extended the search for differences. The differences I found before I stopped were at 965, 2813, 1902, 3441 and 3446.

If the the HP41 and the HP-12C were using the "smarter" y^x algorithm then I would have thought that there would have been no differences. I haven't identified anything unusual about 39, 965, etc.

The results with the hp-33s were 3.1267E-07 with 204 zeroes for either method of calculating.

I found the reference to the use of the Paranoia program to find that the Model 100 used radix 10. It is too long to reproduce here. You can see it by going to Viktor Toth's site and looking at pages 6 and 17 of TI PPC Notes.

I will look at my TI-85 results next. I will also try to get some TI-82 results but my machine has a balky ENTER switch.
.


#13

Quote:
When we were doing comparative tests on machines back in the 1980's it was said that some machines (I think it was the HP's, but I'm not sure, and the results which follow suggest that the supposition was incorrect) had "smarter" y^x algorithms in the sense that if the exponent was an integer, then the machine did repeated multiplications rather than, it was said, taking the logarithm of y, multiplying by x and taking the antilog. The result quoted above was for a program with either the square root multiplied by itself or for the square root squared with the x^2 function. When I did the problem with the square root raised to the second power using the y^x function the result was 3.2137E-05 with 205 zeroes. I didn't mistype. The thousandths digit is really a three.

I wrote a program which compared the results for each integer calculated using the two methods. When the integer was 39 the result with the x^2 method was zero but was 1E-8 with the y^x method. The results were identical for the other 499 integer values. I extended the search for differences. The differences I found before I stopped were at 965, 2813, 1902, 3441 and 3446.

If the the HP41 and the HP-12C were using the "smarter" y^x algorithm then I would have thought that there would have been no differences. I haven't identified anything unusual about 39, 965, etc.


What is happening here is the same phenomenon that causes the HP-71 not to get the "expected" result in the Savage benchmark.

To take the case of 39. The Y^X function takes the natural log of Y, multiplies it by X and then takes the EXP of that result. This all takes place with 13 digit arithmetic on the HP41, HP15, etc.

Do this same calculation with exact arithmetic using a PC program like Derive, Maple, Mathematica, or such, so that you can be sure you're getting correct results. If you start with the 10 digit SQRT(39), take the 13 digit LN, multiply by 2, take the 13 digit EXP you should get exactly 38.99999999502. Notice the last 3 digits. Those digits will be dropped when the result is rounded to 10 digits, but they determine which way the rounding will go. If we round 38.99999999502 to 10 digits we get 40.00000000. Why didn't the HP41 get 40.00000000? When the calculation was done exactly, the result was 38.99999999502, but the HP41 must have gotten a 13 digit internal result a little less than 38.99999999500, and therefore it didn't round the result up. If the HP41's internal result was in error by only -3 ULP's, that would account for the error. This is perfectly excusable. It's why the calculator uses 3 more digits internally than are returned to the user. This way the internal errors are thrown away. It's only when the exactly correct 13 digit value is right on the borderline where it will round up or down that this kind of error occurs.

I wrote a Mathematica program to search for all the values resulting from the n SQRT 2 y^x sequence where the last 3 digits of the exactly correct 13 digit value are 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, or 505, with n ranging from 1 to 100. The values are:

 n     Last 3 of 13 digits
39 502
155 500
240 502
279 504
394 496
552 502
705 502
965 501

Any of these could have exhibited the error, but the HP41 often gets it right. But if the error is going to happen, it will be one of these cases where the rounding boundary is very close.

I searched up to n=7000 and found a bunch more possible candidates for the error. The only way to be sure is to test them all on the HP41 to see if the error occurs. The additional ones I found that make the error are:

4655
4659
5414
6221
6528
6982

And, note that the error the HP41 makes isn't always negative; it can be in either direction. For example, the exact 13 digit result for n=3446 is 3446.000000497 so the HP41's internal result must have had the last 3 digits greater than 500, causing it to round up when it shouldn't have.

This can also happen on a Saturn machine, of course. It will happen when the internal 15 digit result has the last 3 digits very close to 500. I searched for these and got a list of candidates which I then tested on the HP50. The ones that made the error are:

415
519
946
968
979
2715
2718
2730
2772
2798
3151
3447
3475

The HP-33S should make the error with these values of n.

I can't observe what the HP41 is doing internally, but it's pretty easy to see what the internal long real results are on a Saturn machine. So, to show that my explanation is correct, here is what I found on an HP48G using Jim Donnelly's program to do 15 digit arithmetic. The exact result for n=415 should be 415.000000001495, but the HP48G gets 415.000000001502, and in fact when you execute 415 SQRT 2 y^x on the keyboard, you get 415.000000002 instead of 415.000000001. The HP48G got 502 for the last 3 digits of its internal result instead of the 495 it should have gotten, so it rounded up when it shouldn't have.


#14

Rodger:

You wrote about differences between the square root times the square root and the square root squared using the y^x function:

Quote:
.
This can also happen on a Saturn machine, of course. It will happen when the internal 15 digit result has the last 3 digits very close to 500. I searched for these and got a list of candidates which I then tested on the HP50. The ones that made the error are:

415

519

946

968

979

2715

2718

2730

2772

2798

3151

3447

3475

The HP-33S should make the error with these values of n.


I tested my HP-33s (CNA 53200443 in case that makes a difference) using the following program:

LBL R

1

STO B

LBL S

RCL B

SQRT

2

y^x

RCL B

SQRT

ENTER

x

Is x not equal to y?

STOP

1

STO+ B

GTO S

The program stopped at 2798 which is in your list above. It did not stop at any other integer in your list. It also stopped at 5670, 6536 and 6833. I continued to run until the test integer was over 100,000 and did not get any more stops.

Palmer


#15

Quote:
The program stopped at 2798 which is in your list above. It did not stop at any other integer in your list. It also stopped at 5670, 6536 and 6833. I continued to run until the test integer was over 100,000 and did not get any more stops.

Did it stop at any value of n between 1 and 3500 other than at 2798?

I generated the list of possible candidate values up to 100000 and checked the last few. The HP50 failed on the 3rd from the end, namely 99590, and another picked at random from the list, 87778. I'm sure it will fail for a lot more from the list.

Somehow I was thinking the HP-33S was a Saturn based machine. Apparently not.


#16

Quote:


Did it stop at any value of n between 1 and 3500 other than at 2798?


It did not. Just to be sure that I didn't have some sort of curious programming error I checked all of the other numbers on your list between 1 and 3500 from the keyboard. They all yield identical results for the two methods.

I checked a few of the numbers on my second hp33s S/N 52402428 . The results were the same.


Possibly Related Threads...
Thread Author Replies Views Last Post
  Testing card reader motor for 65 Michae Altmann 2 1,092 03-17-2013, 09:24 PM
Last Post: Randy
  [WP 34S] New GCC - Testing required Marcus von Cube, Germany 3 1,021 01-07-2013, 01:59 PM
Last Post: Marcus von Cube, Germany
  Testing the water Ken McPherson 0 585 08-07-2012, 09:18 PM
Last Post: Ken McPherson
  NCEES and Computer Based Testing Richard Garner 2 877 04-10-2012, 07:14 PM
Last Post: Steve Fennell
  WP 34S: Memory Allocation - New Document / Testing Marcus von Cube, Germany 0 562 11-09-2011, 11:48 AM
Last Post: Marcus von Cube, Germany
  Fully testing HP 41 Le Babou 7 1,512 10-23-2011, 08:59 AM
Last Post: Allen
  WP 34S: Some testing required Marcus von Cube, Germany 27 5,101 10-15-2011, 11:15 AM
Last Post: Marcus von Cube, Germany
  OT: Help with Testing Random Number Generators Namir 5 1,357 08-01-2011, 11:20 PM
Last Post: Paul Dale
  Testing functionality of older HP calculators Pat Slattery 2 799 05-25-2011, 09:18 PM
Last Post: Pat Slattery
  OT: CP/M-68K for HP 9000 Series 200 computers Juergen Keller 5 1,142 09-22-2010, 10:09 AM
Last Post: Olivier De Smet

Forum Jump: