Application and integration of computer vision technologies for automated recognition and recording of chemical experiments

Advancements in computer vision have significantly improved motion and object recognition accuracy. These advancements should aid the automatic recognition of chemical experiments, potentially contributing to the recording of experiments. Creating an electronic laboratory notebook from experiment filming enhances convenience and allows more detailed information storage compared to traditional manual recording methods. Our previous research focused on employing object detection and action recognition to automate the recognition of chemical experiments. This paper presents a novel system that combines object detection, action recognition, multiple object tracking, and barcode recognition to automatically generate experimental flowcharts. We implemented our system as a graphical user interface-based application for laboratory use that successfully constructs flowcharts from videos of chemical experiments, including simple chemical manipulations.