Method and system to compensate for inaccuracy associated with processing values with finite precision

A method and system to compensate for inaccuracy associated with processing values with finite precision includes a process for selecting a display value whereby an initial value is provided in a first numbering system. The initial value is then converted into an equivalent stored value in a second...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: HOLLADAY WALTER, SCALORA MICHAEL AMORE, DONG YULIN
Format: Patent
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:A method and system to compensate for inaccuracy associated with processing values with finite precision includes a process for selecting a display value whereby an initial value is provided in a first numbering system. The initial value is then converted into an equivalent stored value in a second numbering system. Then a display value in the first numbering system is determined and selected such that the selected display value includes the least number of significant digits that convert from the first numbering system to the second numbering system exactly as the stored value. The selected display value in the first numbering system is then used for display and/or further processing when the stored value in the second numbering system is recalled.