EXPLAINING MACHINE LEARNING OUTPUT IN INDUSTRIAL APPLICATIONS

There is provided an explainer system (100) for explaining output of a prediction system (10). The prediction system (10) comprises a system-monitor machine learning model (16) trained to predict states of a monitored system (12). The explainer system comprises a perturbator (102) configured to apply predetermined perturbations to original sample data (120, 122) collected from the monitored system (12) to produce perturbed sample data (108, 130), the explainer system being configured to input the perturbed sample data (108, 130) to the prediction system (10). The explainer system further comprises a tester (104) configured to receive model output from the prediction system (10), the model output comprising original model output (110) produced by the system-monitor machine learning model (16) based on the original sample data (120, 122) and deviated model output (126) produced by the system-monitor machine learning model (16) based on the perturbed sample data (108, 130), the deviated model output (126) comprising deviations from the original model output (110), the deviations resulting from the applied perturbations. The explainer system further comprises an extractor (106) configured to receive data (128) defining the perturbations and the resulting deviations and to extract therefrom important features (124) for explaining the model output.