Spark-based Cassandra data import method and device, equipment and medium
The invention discloses a Cassandra data import method based on Spark. The Cassandra data import method comprises the following steps of: importing Cassandra data into a database; relating to the technical field of data processing, the method and the device are used for solving the problems that whe...
Gespeichert in:
1. Verfasser: | |
---|---|
Format: | Patent |
Sprache: | chi ; eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | The invention discloses a Cassandra data import method based on Spark. The Cassandra data import method comprises the following steps of: importing Cassandra data into a database; relating to the technical field of data processing, the method and the device are used for solving the problems that when data is imported into Cassandra through Spark at present; the method comprises the following stepsthat the data size of data to be imported and the size of an SSTable single file are obtained, and the number N of needed partitions is calculated according to the data size and the size of the SSTable single file; calculating a token value according to the Key of the data; according to the token value, allocating the data to the N partitions, and sorting the data; a CQLSSTableWriter is used forreading the sorted data, and an SSTable file is generated; and the SSTable files are processed in parallel, and the SSTable files are imported into a Cassandra cluster through an SSTable. The invention further discloses a Cass |
---|