Method for accelerating loading of data from openGauss to DataFrame
The invention discloses a method for accelerating loading of data from openGauss to DataFrame, and relates to the technical field of computer data processing. According to the method, a connection field conn, an SQL (Structured Query Language) query statement and a protocol field matched with openGa...
Gespeichert in:
Hauptverfasser: | , , , , , |
---|---|
Format: | Patent |
Sprache: | chi ; eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | The invention discloses a method for accelerating loading of data from openGauss to DataFrame, and relates to the technical field of computer data processing. According to the method, a connection field conn, an SQL (Structured Query Language) query statement and a protocol field matched with openGauss are introduced, so that a local computer is connected to an openGauss database management system through ConnectorX; the workflow of the ConnectorX comprises the following steps: obtaining metadata from openGauss according to an SQL (Structured Query Language) query statement; performing source partitioning according to the metadata type, and pre-allocating a memory to create a plurality of NumPy arrays; splitting the SQL query into a plurality of sub-queries, and executing the sub-queries in parallel to obtain a plurality of query results; converting the query result into query result data to be written; and writing the query result data to be written into the NumPy array, and generating the DataFrame. Accordi |
---|