Data management method and system based on kafka, electronic equipment and storage medium

The invention provides a data management method and system based on kafka, electronic equipment and a storage medium, and belongs to the field of distributed data processing. The method comprises the following steps: creating a plurality of kafka theme queues with video object metadata; establishing a plurality of metadata cache queues and kafka partition detection thread services; reading the target video object metadata from the partition of the kafka theme queue through a metadata management service; inserting the target video object metadata into a corresponding metadata cache queue, and inserting a record in a kafka partition detection thread service at the same time; obtaining a video object in a to-be-migrated state from the metadata cache queue according to a migration rule and migrating the video object; a migration result is sent to metadata management, and a partition detection table is updated; and periodically scanning the partition detection table through the kafka partition detection thread ser