John Smitherman posted,

Quote:

Enter 1,000,000 , 1,000,001 and 1,000,002 and find the sample standard deviation. The correct answer should be 1 but as has been discussed before the 33s provided an error message.

Bill Platt replied,

Quote:

This is beyond the range of the machine. It doesn't work on a 32sii nor on a 15c either. It is not a flaw.

Bill is right; the "STAT ERROR" or "Error 2" message is attributable to roundoff errors in the statistical summation. What's ironic is that the failure to produce a result was made possible by *extended-precision calculation*, albeit using standard-precision inputs.

Here are the tabulations:

Xi (Xi)^2 (exact) (Xi)^2 (result retaining 10 or 12 significant digits)
1,000,000 1,000,000,000,000 1,000,000,000,000

1,000,001 1,000,002,000,001 1,000,002,000,000

1,000,002 1,000,004,000,004 1,000,004,000,000

Summations:

S(X) S(X^2) (exact) S(X^2) (using 10 or 12-digit results)

3,000,003 3,000,006,000,005 3,000,006,000,000

([S(X)]^2) (exact result obtained using 13- or 15-digit extended precision)

9,000,018,000,009

([S(X)]^2)/3 (exact result obtained using 13- or 15-digit extended precision)

3,000,006,000,003

The exact calculation

sqrt ([S(X^2) - ([S(X)]^2)/n] / [n-1]) of sample standard deviation is

sqrt [(3,000,006,000,00__5__ - 3,000,006,000,003) / (3-1)] = 1

The "erroneous" calculation is based on 13- or 15-digit internal computaion that utilizes the S(X^2) summations that were rounded off after each datum entry:

sqrt [(3,000,006,000,00__0__ - 3,000,006,000,003) / (3-1)] = ERROR

(The HP-15C's extended-precision calculations for statistical summations are utilized within an example in the HP-15C Advanced Functions Handbook listed on pages 208-211.)

If extended precision had __not__ been employed for the problem discussed here, __both__ terms would have been rounded to 3,000,006,000,000 -- yielding the incorrect result of zero, instead of an error.

-- KS

*Edited: 26 Sept 2007, 2:09 a.m. after one or more responses were posted*