HP Prime numerical precision in CAS and HOME  Printable Version + HP Forums (https://archived.hpcalc.org/museumforum) + Forum: HP Museum Forums (https://archived.hpcalc.org/museumforum/forum1.html) + Forum: Old HP Forum Archives (https://archived.hpcalc.org/museumforum/forum2.html) + Thread: HP Prime numerical precision in CAS and HOME (/thread255657.html) 
HP Prime numerical precision in CAS and HOME  Javier Goizueta  11122013 One funny thing I've noticed about the Prime: the HOME and CAS worlds differ in its internal numerical precision (when CAS operates in approximate mode): In Home the 12decimal digit precision of former HP models is maintained:
1(4/31)*3 => .00000000001 But in the CAS, 47 internal bits of precision is used:
1.0(4./3.1.)*3. => 1.42108547152E14 Which is 2^46. I'd had expected 53 bits (for IEEE 64bit floating point). Anybody can offer more insight into this?
Edited: 12 Nov 2013, 1:17 p.m.
Re: HP Prime numerical precision in CAS and HOME  Joe Horn  11122013 Quote:Prime's CAS uses 48 bits for the mantissa of its floatingpoint numbers. Here's an example using 16 nine's after the decimal point: logb(1.9999999999999999,2) > 48 FWIW, CAS performs its calculations to 48 *truncated* bits, not 48 rounded bits. You're right about Home: That's traditional HP 12digitmantissa BCD.
Joe
Re: HP Prime numerical precision in CAS and HOME  parisse  11132013 Giac uses 48 bits mantissa from the 53 bits from IEEE double. The reason is that Giac stores CAS data (gen type) in 64 bits and 5 bits are used for the data type (24 types are available). We therefore loose 5 bits (the 5 low bits are reset to 0 when a double is retrieved from a gen).
Re: HP Prime numerical precision in CAS and HOME  Ralf Fritzsch  11152013 Should this be the reason behind the following behaviour of CAS.ifactor when used in the HOME screen? CAS.ifactor(17*16!) 2^15*3^6*5^3*7^2*11*13*17 CAS.ifactor(17!) "ifactor(!(17)) Error: Bad Argument Value"
(16! < 2^48 < 17!) Edited: 15 Nov 2013, 7:28 a.m.
Re: HP Prime numerical precision in CAS and HOME  parisse  11152013 No, this is related to limited precision (BCD) floats in Home, not in CAS.
Re: HP Prime numerical precision in CAS and HOME  Paul Dale  11162013 An interesting restriction. Why not grab one bit from the exponent? Does the result get rounded before truncation? Any reason for supporting both decimal and binary arithmetic? HP's decimal reals have some space clear in the exponent and with some packing could easily have five bits clear.
