Prompt-OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision-Language Model Adaptation
Xiwen Chen, Wenhui Zhu, Peijie Qiu, Hao Wang, Huayu Li, Haiyu Wu, Aristeidis Sotiras, Yalin Wang and Abolfazl Razi
视觉语言模型(VLMs),如CLIP,表现出强劲的性能,但在适应下游任务时会挣扎。 及时学习已成为一种高效和有效的策略,可以适应VLM,同时保持其预先训练的知识。 然而,现有方法仍然会导致过拟合和降低零射通概括。 为了应对这一挑战,我们提出了一个最佳的传输(OT)引导的提示学习框架,通过保持预训练和微调模型之间特征分布的结构一致性来缓解遗忘。 与传统的点向约束不同,OT自然地捕获交叉实例关系,并扩展了用于快速调整的可行参数空间,从而在适应和泛化之间实现更好的权衡。 我们的方法对视觉和文本表示都强制了联合约束,确保了整体特征对齐。 基准数据集的广泛实验表明,我们简单而有效的方法可以在基础到新颖的泛化,跨数据集评估和域泛化方面超越现有的快速学习策略,而无需额外的增强或集成技术。 代码可在https://github.com/ChongQingNoSubway/Prompt-OT上查阅。
Vision-language models (VLMs) such as CLIP demonstrate strong performance but struggle when adapted to downstream tasks. Prompt learning has emerged as an efficient and effective strategy to adapt VLMs while preserving their pre-trained knowledge. However, existing methods still lead to overfitting and degrade zero-shot generalization. To address this challenge, we propose an optimal transport (OT)-guided prompt learning framework that mitigates forgetting by preserving the structural consistenc...