Data synchronization quality guarantee method and device based on configuration file
The invention discloses a data synchronization quality guarantee method and device based on a configuration file, and the method comprises the steps: an actuator analyzes a defined Json format configuration file, extracts the connection information of a source data source and a target data source, l...
Gespeichert in:
Hauptverfasser: | , |
---|---|
Format: | Patent |
Sprache: | chi ; eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | The invention discloses a data synchronization quality guarantee method and device based on a configuration file, and the method comprises the steps: an actuator analyzes a defined Json format configuration file, extracts the connection information of a source data source and a target data source, loads a connector corresponding to the source data source and the target data source, and builds the connection with the source data source and the target data source; in a source data source connector and a target data source connector, data of the source data source and data of the target data source are read through a Spark SQL and a DataFrame API and converted into DataFrame type data; calling a source data source connector and a target data source connector in an actuator, acquiring DataFrame type data of the source data source and the target data source, and registering the DataFrame type data as a temporary table of a Spark SQL (Structured Query Language); comparing data differences between the source and the |
---|