Routesplain: Towards Faithful and Intervenable Routing for Software-related Tasks
Adam Štorek, Vikas Upadhyay, Marianne Menglin Liu, Daniel W. Peterson, Anshul Mittal, Sujeeth Bharadwaj, Fahad Shah, Dan Roth
LLM现在处理各种与软件相关的任务,但我们表明,在这些任务中和这些任务中,它们的性能差异很大。 因此,将用户查询路由到适当的LLM可以帮助提高响应质量,同时降低成本。 然而,之前的工作主要集中在通过黑箱模型进行通用的LLM路由。 我们介绍了Routesplain,这是第一个用于软件相关任务的LLM路由器,包括多语言代码生成和修复,输入/输出预测和计算机科学QA。 与现有的路由方法不同,Routesplain首先从每个查询(例如,任务,域,推理复杂性)中提取人类可解释的概念,并且仅基于这些概念的路由,从而提供可理解的,忠实的理由。 我们在8个软件相关任务中评估16个最先进的LLM的Routesplain;Routesplain在准确性和成本方面优于单个模型,等于或超过所有黑箱基线,概念级干预突出了进一步改进路由器的途径。
LLMs now tackle a wide range of software-related tasks, yet we show that their performance varies markedly both across and within these tasks. Routing user queries to the appropriate LLMs can therefore help improve response quality while reducing cost. Prior work, however, has focused mainly on general-purpose LLM routing via black-box models. We introduce Routesplain, the first LLM router for software-related tasks, including multilingual code generation and repair, input/output prediction, and...