Multi-modal fusion for sensing-aided beam tracking in mmWave communications

Millimeter wave (mmWave) communication has attracted extensive attention and research due to its wide bandwidth and abundant spectrum resources. Effective and fast beam tracking is a critical challenge for the practical deployment of mmWave communications. Existing studies demonstrate the potential of sensing-aided beam tracking. However, most studies are focused on single-modal data assistance without considering multi-modal calibration or the impact of inference latency of different sub-modules. Thus, in this study, we design a decision-level multi-modal (mmWave received signal power vector, RGB image and GPS data) fusion for sensing-aided beam tracking (DMBT) method. The proposed DMBT method includes three designed mechanisms, namely normal prediction process, beam misalignment alert and beam tracking correction. The normal prediction process conducts partial beam training instead of exhaustive beam training, which largely reduces large beam training overhead. It also comprehensively selects prediction results from multi-modal data to enhance the DMBT method robustness to noise. The beam misalignment alert based on RGB image and GPS data detects whether there exists beam misalignment and also predicts the optimal beam. The beam tracking correction is designed to capture the optimal beam if misalignment happens by reusing certain blocks in normal prediction process and possibly outdated prediction results. Finally, we evaluate the proposed DMBT method in the vehicle-to-infrastructure scenario based on a real-world dataset. The results show that the method is capable of self-correction and mitigating the negative effect of the relative inference latency. Moreover, 75%–93% beam training overhead can be saved to maintain reliable communication even when faced with considerable noise in measurement data. [Display omitted] •Combining prediction from multi-modal data improves noise resistance ability.•Partial beam training is enough in V2I communication with proposed algorithm.•Outdated prediction is useful for beam tracking correction.•Proposed algorithm adapts different relative inference latency among sub-modules.•Simply increasing beam training overhead isn’t the best when faced with noisy data.