Page data processing method and device, computer equipment and storage medium

The embodiment of the invention belongs to the field of big data, is applied to the field of smart cities, and relates to a page data processing method. The method comprises the following steps of: when a selection instruction sent by a terminal is received, selecting a crawler operator from data pr...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
1. Verfasser: JIA BOTAO
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The embodiment of the invention belongs to the field of big data, is applied to the field of smart cities, and relates to a page data processing method. The method comprises the following steps of: when a selection instruction sent by a terminal is received, selecting a crawler operator from data processing operators deployed by an ETL platform; obtaining crawler configuration information according to a configuration instruction triggered in a crawler configuration page of the terminal; configuring the crawler operator according to the crawler configuration information; running the configuredcrawler operator through a crawler application, and indicating the crawler application to store crawled page data in Redis; adding the page data in the Redis to an ETL data stream of the ETL platform;and performing ETL processing on the ETL data stream to obtain inventory data. The invention further provides a page data processing device, computer equipment and a storage medium. In addition, theinvention further relates t