Implementation method and system of efficient IP proxy pool and data acquisition method

The invention relates to the technical field of IP agents, in particular to an implementation method and system of an efficient IP agent pool, a data acquisition method and an efficient IP agent poolimplementation method. The method comprises the steps: reading a to-be-crawled URL, arranging the to-...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: LI GUANG, ZHU XIANSHEN
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention relates to the technical field of IP agents, in particular to an implementation method and system of an efficient IP agent pool, a data acquisition method and an efficient IP agent poolimplementation method. The method comprises the steps: reading a to-be-crawled URL, arranging the to-be-crawled URL into a task, and adding the task into a task queue; collecting crawlers into the task queue to obtain tasks and execute the tasks, and storing the obtained proxy IPs into an initial queue; verifying the proxy IPs in the initial queue, filtering the unavailable proxy IPs, and adding the available proxy IPs into an effective queue; periodically acquiring the proxy IP from the effective queue, storing the proxy IP into a check queue, acquiring the proxy IP from the check queue for checking, and deleting or putting the proxy IP back into the effective queue according to a checking result; and allocating a proxy IP to the client according to the client request. According to the implementation method and s