Index generation method and index generation device based on MapReduce programming architecture
The invention relates to an index generation method and an index generation device based on a MapReduce programming architecture. The index generation method comprises the following steps of: acquiring data, preparing the data into a unified format and storing the prepared data in a record set formu...
Gespeichert in:
Hauptverfasser: | , , |
---|---|
Format: | Patent |
Sprache: | chi ; eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | The invention relates to an index generation method and an index generation device based on a MapReduce programming architecture. The index generation method comprises the following steps of: acquiring data, preparing the data into a unified format and storing the prepared data in a record set formula; carrying out head encapsulation on each data record in the record set; inserting the data records subjected to data encapsulation into an HBase cluster in batch; calling a MapReduce service and an HBase service in an Hadoop cluster and connecting an Solr cluster; carrying out MapReduce operation and submitting an operation index parallel generating task to form a reverse index intermediate file; carrying out Reduce operation to generate a reverse index file; and starting a new Map task for carrying out slit operation on the reverse index file to generate a final index. According to the index generation method and the index generation device, disclosed by the invention, the storage of high-efficiency distributed |
---|