Spark performance prediction method and device, model training method and device, equipment and medium

The invention relates to the technical field of Spark performance prediction, and provides a Spark performance prediction method and device, a model training method and device, equipment and a medium, and the method comprises the steps: collecting historical data of an application program running in...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: ZHANG CHENG, SHEN CHAO, ZHAO JIE, LI XIAOLI, ZHONG HAN, CHEN CHEN, HAN YUANBIN
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention relates to the technical field of Spark performance prediction, and provides a Spark performance prediction method and device, a model training method and device, equipment and a medium, and the method comprises the steps: collecting historical data of an application program running in a Spark system; preprocessing the historical data to obtain a processed data set; the processed data set is adopted to train a Spark performance prediction model; wherein the Spark performance prediction model comprises a multi-head attention network and a convolutional neural network, and the multi-head attention network is used for extracting semantic information of input data and operation features; the convolutional neural network is used for extracting spatial correlation features of the input data and the job features; predicting performance indexes of application program operation in the Spark system according to the semantic information and the spatial correlation characteristics, and outputting a predicti