Data processing method and system based on data lake

The invention discloses a data processing method and system based on a data lake, which is applied to a platform comprising a data warehouse, and comprises the following steps: classifying all source data information based on a data access specification, constructing a source pasting table on a sour...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: HAN LIANG, XU YINLING, LIU LUQING, HAN JIEJIAO, WU JIALE, DU WANBO, MENG ZIHAN, CHEN JIA
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention discloses a data processing method and system based on a data lake, which is applied to a platform comprising a data warehouse, and comprises the following steps: classifying all source data information based on a data access specification, constructing a source pasting table on a source pasting layer, and importing a data source file into the data lake; the method comprises the following steps: analyzing a service demand according to a service application, carrying out dimension modeling based on the service demand, creating a dimension table and a fact table, setting a data index according to the fact table, and establishing a market special topic at a market layer based on the data index; checking the field needing to be monitored of the dimension modeling from the source pasting table; constructing a summary table in a summary layer, and performing metadata acquisition and monitoring on the dimension table, the fact table and the summary table; and opening the data tables in the summary laye