Distributed data acquisition method and related device
The invention is suitable for the technical field of data acquisition, provides a distributed data acquisition method and a related device, and aims to help a user to improve the data acquisition speed of a target website under the condition of relatively low cost. When the distributed data acquisit...
Gespeichert in:
Hauptverfasser: | , |
---|---|
Format: | Patent |
Sprache: | chi ; eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | The invention is suitable for the technical field of data acquisition, provides a distributed data acquisition method and a related device, and aims to help a user to improve the data acquisition speed of a target website under the condition of relatively low cost. When the distributed data acquisition method is applied to a distributed host, the distributed data acquisition method mainly comprises: determining a data acquisition task, wherein the data acquisition task comprises a target keyword and a target website; splitting the data acquisition task into X data acquisition sub-tasks, wherein X is a positive integer greater than 0; distributing the data acquisition sub-task to a slave in the distributed system, so that the slave returns a corresponding data acquisition sub-result after completing data acquisition of the target keyword in the target website; receiving the data acquisition sub-result; and summarizing the data acquisition sub-results to obtain a data acquisition result.
本申请适用于数据获取技术领域,提供了一种分布式 |
---|