Backdoor Attacks Against Speech Language Models
Alexandrine Fortier, Thomas Thebaud, Jesús Villalba, Najim Dehak, Patrick Cardinal
大型语言模型(LLM)及其多模态扩展越来越受欢迎。 启用多模态的一种常见方法是使用LLM对特定域的编码器进行级联,使生成的模型继承其所有组件的漏洞。 在这项工作中,我们提出了第一个针对语音语言模型的音频后门攻击的系统研究。 我们在四个语音编码器和三个数据集上展示了它的有效性,涵盖了四个任务:自动语音识别(ASR),语音识别以及性别和年龄预测。 该攻击持续获得高成功率,从90.76%到99.41%不等。 为了更好地了解后门的传播方式,我们进行了组件分析,以确定管道中最脆弱的阶段。 最后,我们提出了一个基于微调的防御,以减轻中毒预训练编码器的威胁。
Large Language Models (LLMs) and their multimodal extensions are becoming increasingly popular. One common approach to enable multimodality is to cascade domain-specific encoders with an LLM, making the resulting model inherit vulnerabilities from all of its components. In this work, we present the first systematic study of audio backdoor attacks against speech language models. We demonstrate its effectiveness across four speech encoders and three datasets, covering four tasks: automatic speech ...