Distributed storage and parallel calculation-based power grid data quality detection method

The invention discloses a distributed storage and parallel calculation-based power grid data quality detection method, which comprises the following steps of storing an original data record by adopting an HBase; establishing a query index for a field related to a checking rule by adopting the HBase;...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: LONG QINGLIN, CHEN CHENGZHI, LIANG GUOHUI, HUANG YIHUA, GU RONG, YANG BINCHENG
Format: Patent
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention discloses a distributed storage and parallel calculation-based power grid data quality detection method, which comprises the following steps of storing an original data record by adopting an HBase; establishing a query index for a field related to a checking rule by adopting the HBase; establishing a timestamp index for the original data record so as to provide support for incremental data quality checking and small-time granularity data quality checking by adopting the HBase; storing an auxiliary index file and an operation log file of the data record so as to rapidly load checking data and improve checking performance during total historical data quality checking by adopting an HDFS (hadoop distributed file system); performing MapReduce-based checking rule parallel processing to improve the checking performance. According to the method, the problems of poor extensibility, long checking time delay and low system cost performance of a conventional relational database system-based power grid data quality detection method are solved.