Taking the next step with generative artificial intelligence: The transformative role of multimodal large language models in science education

The integration of Artificial Intelligence (AI), particularly Large Language Model (LLM)-based systems, in education has shown promise in enhancing teaching and learning experiences. However, the advent of Multimodal Large Language Models (MLLMs) like GPT-4 Vision, capable of processing multimodal data including text, sound, and visual inputs, opens a new era of enriched, personalized, and interactive learning landscapes in education. This paper derives a theoretical framework for integrating MLLMs into multimodal learning. This framework serves to explore the transformative role of MLLMs in central aspects of science education by presenting exemplary innovative learning scenarios. Possible applications for MLLMs range from content creation to tailored support for learning, fostering engagement in scientific practices, and providing assessments and feedback. These applications are not limited to text-based and uni-modal formats but can be multimodal, thus increasing personalization, accessibility, and potential learning effectiveness. Despite the many opportunities, challenges such as data protection and ethical considerations become salient, calling for robust frameworks to ensure responsible integration. This paper underscores the necessity for a balanced approach in implementing MLLMs, where the technology complements rather than supplants the educators' roles, ensuring an effective and ethical use of AI in science education. It calls for further research to explore the nuanced implications of MLLMs for educators and to extend the discourse beyond science education to other disciplines. Through developing a theoretical framework for the integration of MLLMs into multimodal learning and exploring the associated potentials, challenges, and future implications, this paper contributes to a preliminary examination of the transformative role of MLLMs in science education and beyond. •Introduces a framework for incorporating Multimodal Large Language Models (MLLMs) into (science) education•Highlights MLLMs' ability to dynamically adjust educational content across various formats, improving accessibility and engagement for diverse learning needs.•Discusses how MLLMs can foster learning by providing various examples of potential integration in science education where understanding complex concepts often requires integrating multiple types of information.•Shows how MLLMs can tailor learning experiences to meet individual needs, aligning with the journal’s emphasi