Batch data processing method and device, electronic device and storage medium
The invention relates to a batch data processing method and device, an electronic device and a storage medium, and the method comprises the steps: transmitting real-time data in a relational database to a message queue based on a configuration file, and obtaining to-be-processed data in the message...
Gespeichert in:
Hauptverfasser: | , , , , , |
---|---|
Format: | Patent |
Sprache: | chi ; eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | The invention relates to a batch data processing method and device, an electronic device and a storage medium, and the method comprises the steps: transmitting real-time data in a relational database to a message queue based on a configuration file, and obtaining to-be-processed data in the message queue; constructing a data import task file based on the data format of the to-be-processed data, the basic information of the message queue, the category name of the to-be-processed data and the number of partitions of the to-be-processed data; based on the data import task file, importing the to-be-processed data into a data table in a target database to generate a data import table, the target database being a StarRocks database; and performing data processing on the data in the data import table based on the data processing script file to obtain a real-time data processing result. By means of the method and device, the problem that the flexibility of batch data calculation in the database is low is solved, and |
---|