Spark-based Cassandra data import method and device, equipment and medium

The invention discloses a Cassandra data import method based on Spark. The Cassandra data import method comprises the following steps of: importing Cassandra data into a database; relating to the technical field of data processing, the method and the device are used for solving the problems that whe...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
1. Verfasser: CHENG WANSHENG
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention discloses a Cassandra data import method based on Spark. The Cassandra data import method comprises the following steps of: importing Cassandra data into a database; relating to the technical field of data processing, the method and the device are used for solving the problems that when data is imported into Cassandra through Spark at present; the method comprises the following stepsthat the data size of data to be imported and the size of an SSTable single file are obtained, and the number N of needed partitions is calculated according to the data size and the size of the SSTable single file; calculating a token value according to the Key of the data; according to the token value, allocating the data to the N partitions, and sorting the data; a CQLSSTableWriter is used forreading the sorted data, and an SSTable file is generated; and the SSTable files are processed in parallel, and the SSTable files are imported into a Cassandra cluster through an SSTable. The invention further discloses a Cass