HP Forums

Full Version: RM mode and its effects on RRM determinant of AM1 matrix
You're currently viewing a stripped down version of our content. View the full version with proper formatting.

RM Mode       Determinant
------- ---------------------
0 0.99999 711935 38228
1 0.99999 720675 30552
2 0.99999 709507 09280
3 1.00000 009178 43880
4 0.99999 711925 74617
5 1.00000 895891 46320
6 0.99999 718613 18536

Looks like RM 3 is closest to 1. Fairly significantly closer, it appears.

RM 3 is round half up. It being closer is most likely a fluke.

This does show that the algorithm isn't overly sensitive to rounding mode which is goodness.


- Pauli