Data synchronization quality guarantee method and device based on configuration file

The invention discloses a data synchronization quality guarantee method and device based on a configuration file, and the method comprises the steps: an actuator analyzes a defined Json format configuration file, extracts the connection information of a source data source and a target data source, l...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: YAN YISHUAI, MAO CHUNYANG
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention discloses a data synchronization quality guarantee method and device based on a configuration file, and the method comprises the steps: an actuator analyzes a defined Json format configuration file, extracts the connection information of a source data source and a target data source, loads a connector corresponding to the source data source and the target data source, and builds the connection with the source data source and the target data source; in a source data source connector and a target data source connector, data of the source data source and data of the target data source are read through a Spark SQL and a DataFrame API and converted into DataFrame type data; calling a source data source connector and a target data source connector in an actuator, acquiring DataFrame type data of the source data source and the target data source, and registering the DataFrame type data as a temporary table of a Spark SQL (Structured Query Language); comparing data differences between the source and the