Fine-Tuning Vision-Language Models for Multimodal Polymer Property Prediction
An Vuong, Minh-Hao Van, Prateek Verma, Chen Zhao and Xintao Wu
视觉语言模型(VLM)在视觉问题回答和多模态文本生成等任务中表现出强劲的表现,但它们在材料科学等科学领域的有效性仍然有限。 虽然一些机器学习方法已经解决了这一领域的特定挑战,但仍然缺乏为使用多模态数据进行聚合物属性预测等广泛任务设计的基础模型。 在这项工作中,我们介绍了一个多模态聚合物数据集,通过指令调整对微调VLM,并评估多模态对预测性能的影响。 我们的微调模型,使用LoRA,优于单模态和基线方法,展示了多模态学习的好处。 此外,这种方法减少了为不同属性训练不同模型的需求,降低了部署和维护成本。
Vision-Language Models (VLMs) have shown strong performance in tasks like visual question answering and multimodal text generation, but their effectiveness in scientific domains such as materials science remains limited. While some machine learning methods have addressed specific challenges in this field, there is still a lack of foundation models designed for broad tasks like polymer property prediction using multimodal data. In this work, we present a multimodal polymer dataset to fine-tune VL...