Information processing method and system based on big data

The invention discloses an information processing method and system based on big data, belongs to the technical field of Internet of Things, is used for multi-source heterogeneous data processing of an Internet of Things scene, and comprises the following steps: constructing a data acquisition pipeline, a data preprocessing pipeline, a feature engineering pipeline, a model training pipeline and an assembly line based on Kubernetes; structured data and unstructured data are collected from multiple sources in parallel, features are extracted from the unstructured data, all the collected data are preprocessed, different CNN and RNN models are trained in parallel, integrated learning and incremental learning are used for model optimization, key nodes and monitoring indexes are set, and a system is optimized in a closed-loop mode according to the indexes. Aiming at the problem of low model precision in the prior art, massive, heterogeneous and dynamically changing data in the Internet of Things scene can be effici