A Slim Prompt-Averaged Consistency prompt learning for vision–language model

Recent advancements in prompt tuning have enhanced the adaptation of large pre-trained models to target tasks. However, existing methods struggle to establish an effective balance between task-specific knowledge and generalizable knowledge during tuning, skewing too heavily towards one at the expense of the other. To address this issue, we propose a Slim Prompt-Averaged Consistency (SPAC) prompt learning approach. Specifically, SPAC introduces a temporal ensembling-based averaged-prompt module and leverages a multifaceted consistency mechanism to ensure knowledge consistency under the guidance of averaged-prompt. Additionally, SPAC employs the contrastive learning strategy to further enhance the learning of target task representations based on positive and negative sample pairs. Furthermore, considering the notable resource consumption of existing prompt formats, we refine the prompt format, significantly reducing resource consumption during training and inference. Extensive experiments on 11 benchmark datasets demonstrate that our approach outperforms others in few-shot prompt learning transfer tasks, including base-to-novel generalization and cross-dataset transfer, while consuming fewer resources.