Posts: 117

Threads: 31

Joined: Sep 2008

Statistics subsystem does not like certain numbers in Y. For example,

350.33050223 ENTER

3.265 [+/-] [EEX] 10 [+/-]

[+]

Cl[SIGMA]

1 [SIGMA]+

2 [SIGMA]+

3 [SIGMA]+

4 [SIGMA]+

5 [SIGMA]+

[g] [5]

Attempt to calculate s (standard deviation) gives domain error.

This time, the effect is reproducible on emulator.

This is highly number-specific. Many numbers are breaking it, but many numbers pass through. Without that -3.265E-10 it does not give error.

Posts: 4,587

Threads: 105

Joined: Jul 2005

FWIW, it does return 1.581 and not give any error on the real thing (2.2 2738).

d:-)

Posts: 3,229

Threads: 42

Joined: Jul 2006

For each accumulation you made, the value in Y was unchanged so s_{y} should be zero.

However, the problem is due to a rounding error during the calculation of s_{y}. Sigma y^{2} - n * (sigma y)^{2} ends up being slightly negative.

I'd have preferred to use the robust formulas for this but was hamstrung by the need for backwards compatibility.

I've committed a fix which will be in the next firmware revision.

- Pauli

*Edited: 19 June 2013, 2:08 a.m. *

Posts: 117

Threads: 31

Joined: Sep 2008

And what are the robust formulas?

And why they are not backward compatible?

Posts: 3,229

Threads: 42

Joined: Jul 2006

Formulas for variance. Specifically, the on-line algorithm.

Recovering the various summation terms we had to support isn't trivial -- people expect the usual sums: x, x^{2}, y, y^{2}, xy and all the logarithmic variations. Far easier to just use the traditional method, albeit in double precision for some of the summations.

- Pauli

Posts: 2,247

Threads: 200

Joined: Jun 2005

Thanks for the link Paul. I have been looking for robust algorithms to calculate the mean and standard deviation.