Efficient distributed monitoring with active Collaborative Prediction

Isolating users from the inevitable faults in large distributed systems is critical to Quality of Experience. We formulate the problem of probe selection for fault prediction based on end-to-end probing as a Collaborative Prediction (CP) problem. On an extensive experimental dataset from the EGI (European Grid Initiative) grid, the combination of the Maximum Margin Matrix Factorization approach to CP and Active Learning shows excellent performance, reducing the number of probes typically by 80% to 90%. Comparison with other Collaborative Prediction strategies show that Active Probing is most efficient at dealing with the various sources of data variability. •We model probe selection for fault prediction as a Collaborative Prediction task.•Performance improvement over the baseline is demonstrated on a real-world dataset.•Active Probing is required to cope with the various sources of data variability.