Distributed data acquisition method and related device

The invention is suitable for the technical field of data acquisition, provides a distributed data acquisition method and a related device, and aims to help a user to improve the data acquisition speed of a target website under the condition of relatively low cost. When the distributed data acquisit...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: ZENG XIANGSONG, LUO WEIDONG
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention is suitable for the technical field of data acquisition, provides a distributed data acquisition method and a related device, and aims to help a user to improve the data acquisition speed of a target website under the condition of relatively low cost. When the distributed data acquisition method is applied to a distributed host, the distributed data acquisition method mainly comprises: determining a data acquisition task, wherein the data acquisition task comprises a target keyword and a target website; splitting the data acquisition task into X data acquisition sub-tasks, wherein X is a positive integer greater than 0; distributing the data acquisition sub-task to a slave in the distributed system, so that the slave returns a corresponding data acquisition sub-result after completing data acquisition of the target keyword in the target website; receiving the data acquisition sub-result; and summarizing the data acquisition sub-results to obtain a data acquisition result. 本申请适用于数据获取技术领域,提供了一种分布式