Method and system to compensate for inaccuracy associated with processing values with finite precision
A method and system to compensate for inaccuracy associated with processing values with finite precision includes a process for selecting a display value whereby an initial value is provided in a first numbering system. The initial value is then converted into an equivalent stored value in a second...
Gespeichert in:
Hauptverfasser: | , , |
---|---|
Format: | Patent |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | A method and system to compensate for inaccuracy associated with processing values with finite precision includes a process for selecting a display value whereby an initial value is provided in a first numbering system. The initial value is then converted into an equivalent stored value in a second numbering system. Then a display value in the first numbering system is determined and selected such that the selected display value includes the least number of significant digits that convert from the first numbering system to the second numbering system exactly as the stored value. The selected display value in the first numbering system is then used for display and/or further processing when the stored value in the second numbering system is recalled. |
---|