HP Prime numerical precision in CAS and HOME « Next Oldest | Next Newest »

 ▼ Javier Goizueta Unregistered Posts: 25 Threads: 9 Joined: Aug 2007 11-12-2013, 01:16 PM One funny thing I've noticed about the Prime: the HOME and CAS worlds differ in its internal numerical precision (when CAS operates in approximate mode): In Home the 12-decimal digit precision of former HP models is maintained: ``` 1-(4/3-1)*3 => .00000000001 ``` But in the CAS, 47 internal bits of precision is used: ``` 1.0-(4./3.-1.)*3. => 1.42108547152E-14 ``` Which is 2^-46. I'd had expected 53 bits (for IEEE 64-bit floating point). Anybody can offer more insight into this? Edited: 12 Nov 2013, 1:17 p.m. ▼ Joe Horn Unregistered Posts: 153 Threads: 7 Joined: Jun 2008 11-12-2013, 11:19 PM Quote: One funny thing I've noticed about the Prime: the HOME and CAS worlds differ in its internal numerical precision (when CAS operates in approximate mode)... Prime's CAS uses 48 bits for the mantissa of its floating-point numbers. Here's an example using 16 nine's after the decimal point: logb(1-.9999999999999999,2) --> -48 FWIW, CAS performs its calculations to 48 *truncated* bits, not 48 rounded bits. You're right about Home: That's traditional HP 12-digit-mantissa BCD. -Joe- parisse Unregistered Posts: 38 Threads: 0 Joined: Aug 2013 11-13-2013, 02:45 PM Giac uses 48 bits mantissa from the 53 bits from IEEE double. The reason is that Giac stores CAS data (gen type) in 64 bits and 5 bits are used for the data type (24 types are available). We therefore loose 5 bits (the 5 low bits are reset to 0 when a double is retrieved from a gen). ▼ Ralf Fritzsch Unregistered Posts: 17 Threads: 0 Joined: Aug 2010 11-15-2013, 07:26 AM Should this be the reason behind the following behaviour of CAS.ifactor when used in the HOME screen? CAS.ifactor(17*16!) 2^15*3^6*5^3*7^2*11*13*17 CAS.ifactor(17!) "ifactor(!(17)) Error: Bad Argument Value" (16! < 2^48 < 17!) Edited: 15 Nov 2013, 7:28 a.m. ▼ parisse Unregistered Posts: 38 Threads: 0 Joined: Aug 2013 11-15-2013, 09:55 AM No, this is related to limited precision (BCD) floats in Home, not in CAS. Paul Dale Unregistered Posts: 3,229 Threads: 42 Joined: Jul 2006 11-16-2013, 03:51 AM An interesting restriction. Why not grab one bit from the exponent? Does the result get rounded before truncation? Any reason for supporting both decimal and binary arithmetic? HP's decimal reals have some space clear in the exponent and with some packing could easily have five bits clear. - Pauli

 Possibly Related Threads… Thread Author Replies Views Last Post HP Prime: CAS taylor Alberto Candel 5 4,011 12-13-2013, 09:45 PM Last Post: Alberto Candel HP Prime CAS curiosity bluesun08 11 4,844 12-10-2013, 01:03 PM Last Post: Han [HP-Prime CAS] "Warning, ^ (Command) Is ambiguous on non square matrices"?? CompSystems 1 2,190 12-07-2013, 07:15 PM Last Post: CompSystems HP Prime: complex numbers in CAS. Alberto Candel 1 1,904 12-06-2013, 02:36 PM Last Post: parisse HP Prime: Proper Use of Home View and CAS View James Williams 9 4,268 12-05-2013, 02:44 PM Last Post: James Williams [HP Prime] plotfunc() bug in CAS Chris Pem10 2 2,410 12-04-2013, 02:46 PM Last Post: Chris Pem10 HP Prime: Home/CAS? Alasdair McAndrew 11 3,900 11-26-2013, 02:48 PM Last Post: Alberto Candel HP Prime numerical restrictions? Alasdair McAndrew 4 1,812 11-16-2013, 05:32 PM Last Post: Alasdair McAndrew Need CAS help Michael de Estrada 6 2,652 11-13-2013, 01:29 AM Last Post: cyrille de BrĂ©bisson HP Prime - CAS functions in Spreadsheet App CR Haeger 6 2,470 11-11-2013, 12:37 AM Last Post: Michael de Estrada

Forum Jump: