BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference
Yun Wang, Lingyun Yang, Senhao Yu, Yixiao Wang, Ruixing Li, Zhixiang Wei, James Yen, and Zhengwei Qi
混合专家(MoE)架构通过为每个输入令牌仅激活一个专门的专家网络子集来扩展语言模型,从而减少浮点运算的数量。 然而,现代MoE模型的不断增长的尺寸导致其完整参数集超过GPU内存容量;例如,Mixtral-8x7B具有450亿个参数,并且需要87 GB内存,即使每个令牌仅使用140亿个参数。 现有系统通过将非活动专家卸载到CPU内存来缓解这一限制,但跨PCIe互连转移专家会产生显着的延迟(约10毫秒)。 预选方法旨在通过预测需要哪些专家来隐藏这种延迟,但预取失败引入了显着的失速并放大了推理延迟。 如果发生预扣故障,前期工作提供了两个主要解决方案:要么按需获取专家,由于PCIe瓶颈而导致长时间失速,要么将专家从计算中退出,这显着降低了模型的准确性。 因此,关键挑战是在预选失败时保持高推理速度和模型精度。
Mixture-of-Experts (MoE) architectures scale language models by activating only a subset of specialized expert networks for each input token, thereby reducing the number of floating-point operations. However, the growing size of modern MoE models causes their full parameter sets to exceed GPU memory capacity; for example, Mixtral-8x7B has 45 billion parameters and requires 87 GB of memory even though only 14 billion parameters are used per token. Existing systems alleviate this limitation by off...