ETL data quality judgment and feedback method

The invention discloses an ETL data quality judgment and feedback method, and the method comprises the steps: obtaining a judgment index of an ETL data processing process, determining a quality judgment result of the ETL data processing process according to a comparison result of a result value of t...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: ZHAO HAIBIN, HAO LIANG, LI YIREN, JIA YONGPO, LIN YATUAN, MING YONGJIE, SHEN PEI, WEI XIAOFEI, XING WEIHAN
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention discloses an ETL data quality judgment and feedback method, and the method comprises the steps: obtaining a judgment index of an ETL data processing process, determining a quality judgment result of the ETL data processing process according to a comparison result of a result value of the judgment index and a preset expected value, and determining that an ETL task fails according to an ETL processing log; determining the dirty data from the data warehouse according to the ETL processing log and the mark information carried in the target data, and cleaning the dirty data, thereby further improving the accuracy of ETL data quality judgment, performing feedback through a quality judgment result, discovering problems in the ETL processing process in time, and ensuring the accuracyof output data. 本发明公开了一种ETL数据质量判定与反馈方法,该方法包括:获取ETL数据处理过程的判定指标,根据所述判定指标的结果值和预设期望值的比较结果确定对所述ETL数据处理过程的质量判定结果,并在根据ETL处理日志确定所述ETL任务失败时,根据所述ETL处理日志和目标数据中携带的标记信息从所述数据仓库中确定出所述脏数据,并对所述脏数据进行清理,从而进一步提高了对ETL数据质量判定的准确性,并通过质量判定结果进行反馈,及时发