![]() |
Rounding errors - Printable Version +- HP Forums (https://archived.hpcalc.org/museumforum) +-- Forum: HP Museum Forums (https://archived.hpcalc.org/museumforum/forum-1.html) +--- Forum: Old HP Forum Archives (https://archived.hpcalc.org/museumforum/forum-2.html) +--- Thread: Rounding errors (/thread-159883.html) |
Rounding errors - BruceH - 11-21-2009 Just to emphasise that rounding errors aren't just limited to calculators here's something culled from the Interweb...
Quote: Re: Rounding errors - Lyuka - 11-21-2009 24.5 days means 2^31 [ms] It is probably an old singed and unsigned or wrap-round bug but not rounding errors.
The source or related article is Slashdot IT Story
Re: Rounding errors - Paul Dale - 11-21-2009 So it isn't so much a rounding error as an integer overflow.
Still, I cannot fathom a reason why the autofocus code would care about the time.
- Pauli
Re: Rounding errors - David Hayden - 11-22-2009 Quote: I have minimal knowledge of how autofocus works, but that never stopped me from speculating before.... :) The routine might do something like this:
- Check to focusNow if you get the time from a millisecond counter and mistakenly interpret it as a signed value, you'll think that the time interval was negative and all hell breaks loose.
Again, this is purely speculation, but I think it illustrates how time might factor into their calculations.
Re: Rounding errors - Vieira, Luiz C. (Brazil) - 11-22-2009 Quote:In this case, we actually have an algorithmic error.
Luiz (Brazil)
|