Batch data processing method and device, electronic device and storage medium

The invention relates to a batch data processing method and device, an electronic device and a storage medium, and the method comprises the steps: transmitting real-time data in a relational database to a message queue based on a configuration file, and obtaining to-be-processed data in the message...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: WANG YUEJIAN, GU WEITAO, LIU PEIKAI, CAO CAIPENG, ZHOU YOU, CHEN FEI
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention relates to a batch data processing method and device, an electronic device and a storage medium, and the method comprises the steps: transmitting real-time data in a relational database to a message queue based on a configuration file, and obtaining to-be-processed data in the message queue; constructing a data import task file based on the data format of the to-be-processed data, the basic information of the message queue, the category name of the to-be-processed data and the number of partitions of the to-be-processed data; based on the data import task file, importing the to-be-processed data into a data table in a target database to generate a data import table, the target database being a StarRocks database; and performing data processing on the data in the data import table based on the data processing script file to obtain a real-time data processing result. By means of the method and device, the problem that the flexibility of batch data calculation in the database is low is solved, and