HP Forums

Full Version: HP Prime: Rounding error in determinant
You're currently viewing a stripped down version of our content. View the full version with proper formatting.

Let's assume the following matrix:

[[4 5 7] [8 9 0] [5 6 2]]

The determinant (as given by my Prime (hardware & emulator, both running version 2013 8 13. Rev:5016)) is 13.

The transposed matrix is:

[[4 8 5] [5 9 6] [7 0 2]]

The corresponding determinant is 13.0000000002

Why? Shouldn't they be the same?

btw: my HP15C also makes a small error (-2e-8) in the determinant of the transposed matrix...

Short Answer: Floating Point Computations (I think)


In CAS mode you will get the same and exact result.

Probably some rounding effect in HOME

By the way the HP50 give 13 both in exact and approx mode

Edited: 25 Oct 2013, 7:16 p.m.

FWIW, the WP 34S returns exact 13 for both the original and its transposed.