Index generation method and index generation device based on MapReduce programming architecture

The invention relates to an index generation method and an index generation device based on a MapReduce programming architecture. The index generation method comprises the following steps of: acquiring data, preparing the data into a unified format and storing the prepared data in a record set formu...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: ZHANG ZHENGDAO, LAN YILUN, TANG WEIBIN
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention relates to an index generation method and an index generation device based on a MapReduce programming architecture. The index generation method comprises the following steps of: acquiring data, preparing the data into a unified format and storing the prepared data in a record set formula; carrying out head encapsulation on each data record in the record set; inserting the data records subjected to data encapsulation into an HBase cluster in batch; calling a MapReduce service and an HBase service in an Hadoop cluster and connecting an Solr cluster; carrying out MapReduce operation and submitting an operation index parallel generating task to form a reverse index intermediate file; carrying out Reduce operation to generate a reverse index file; and starting a new Map task for carrying out slit operation on the reverse index file to generate a final index. According to the index generation method and the index generation device, disclosed by the invention, the storage of high-efficiency distributed