Method for accelerating loading of data from openGauss to DataFrame

The invention discloses a method for accelerating loading of data from openGauss to DataFrame, and relates to the technical field of computer data processing. According to the method, a connection field conn, an SQL (Structured Query Language) query statement and a protocol field matched with openGa...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: PENG ZHIHAO, SU HAN, ZHENG KAI, LIU YUCHI, YUAN ZEWEN, ZENG KAI
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention discloses a method for accelerating loading of data from openGauss to DataFrame, and relates to the technical field of computer data processing. According to the method, a connection field conn, an SQL (Structured Query Language) query statement and a protocol field matched with openGauss are introduced, so that a local computer is connected to an openGauss database management system through ConnectorX; the workflow of the ConnectorX comprises the following steps: obtaining metadata from openGauss according to an SQL (Structured Query Language) query statement; performing source partitioning according to the metadata type, and pre-allocating a memory to create a plurality of NumPy arrays; splitting the SQL query into a plurality of sub-queries, and executing the sub-queries in parallel to obtain a plurality of query results; converting the query result into query result data to be written; and writing the query result data to be written into the NumPy array, and generating the DataFrame. Accordi