DATA PROCESSING WITH INTEGRATED METADATA GENERATION AND STORAGE

A method of generating and storing metadata in a data processing system includes defining metadata sets of process definition metadata based on a process definition of the data processing system. The metadata sets include a first set of metadata corresponding to a processing step of the data processing system, a second set of metadata corresponding to a processing step successor, and a third set of metadata corresponding to a data object that is produced or consumed by the processing step in the data processing system. The method further includes executing one or more steps of the data processing system according to the process definition and generating runtime metadata during an execution of the one or more data processing system steps. The method further includes storing the runtime metadata and forming a metadata data store that integrates the runtime metadata and the metadata sets of process definition metadata.