A Historical Interaction-Enhanced Shapley Policy Gradient Algorithm for Multi-Agent Credit Assignment
Ao Ding, Licheng Sun, Yongjie Hou, Huaqing Zhang, Hongbin Ma
多智能体强化学习(MARL)在多剂协作问题中表现出显着的性能,近年来已成为人工智能研究的突出主题。 然而,MARL中传统的信用分配方案在保持训练稳定性的同时,无法可靠地捕获强耦合任务中的单个贡献,从而导致有限的泛化能力并阻碍算法性能。 为了应对这些挑战,我们提出了多代理信用分配的历史交互增强型沙普利策略梯度算法(HIS),该算法采用混合信用分配机制来平衡基础奖励与个人贡献激励。 通过利用历史交互数据以样本效率的方式计算沙普利值,HIS增强了代理感知自身贡献的能力,同时保留了全球奖励以维持训练稳定性。 此外,我们还为混合信用分配机制提供理论保证,确保其产生的赋值结果既高效又稳定。 我们在三个广泛使用的连续动作基准环境中评估拟议的算法:多代理粒子环境,多代理MuJoCo和Bi-DexHands。 实验结果表明,HIS优于最先进的方法,特别是在强耦合,复杂的协作任务方面表现出色。
Multi-agent reinforcement learning (MARL) has demonstrated remarkable performance in multi-agent collaboration problems and has become a prominent topic in artificial intelligence research in recent years. However, traditional credit assignment schemes in MARL cannot reliably capture individual contributions in strongly coupled tasks while maintaining training stability, which leads to limited generalization capabilities and hinders algorithm performance. To address these challenges, we propose ...