Meta CLIP 2: A Worldwide Scaling Recipe
Yung-Sung Chuang, Yang Li, Dong Wang, Ching-Feng Yeh, Kehan Lyu, Ramya Raghavendra, James Glass, Lifei Huang, Jason Weston, Luke Zettlemoyer, Xinlei Chen, Zhuang Liu, Saining Xie, Wen-tau Yih, Shang-Wen Li, Hu Xu
对比语言-图像预训练(CLIP)是一种流行的基础模型，支持从零样本分类、检索到多模态大语言模型(MLLM)编码器等多种任务。尽管CLIP已在英语世界的十亿规模图文对上成功训练，但将其训练规模进一步扩展到全球网络数据仍面临挑战：(1)缺乏处理非英语世界数据的筛选方法；(2)现有多语言CLIP的英语性能低于纯英语版本，即大语言模型(LLM)中常见的"多语言诅咒"。本文提出Meta CLIP 2，首个基于全球网络规模图文对从头训练CLIP的方案。为验证发现的普适性，我们通过最小必要改动进行严格消融实验，提出了一个能使英语和非英语世界数据相互受益的方案。在零样本ImageNet分类任务中，Meta CLIP 2 ViT-H/14比纯英语版本高出0.8个百分点。
Contrastive Language-Image Pretraining (CLIP) is a popular foundation model, supporting from zero-shot classification, retrieval to encoders for multimodal large language models (MLLMs). Although CLIP is successfully trained on billion-scale image-text pairs from the English world, scaling CLIP's training further to learning from the worldwide web data is still challenging: (1) no curation method is available to handle data points from non-English world; (2) the English performance from existing...