AR Secretary Agent: Real-time Memory Augmentation via LLM-powered Augmented Reality Glasses
Raphaël A. El Haddad, Zeyu Wang, Yeonsu Shin, Ranyi Liu, Yuntao Wang, Chun Yu
对于许多专业人士来说,每天与大量个人互动是司空见惯的事情,这可能导致在回顾具体细节时面临挑战:这个人是谁? 我们上次谈了什么? 增强现实(AR)眼镜的优势,配备了视觉和听觉数据采集功能,提出了一个解决方案。 在我们的工作中,我们实施了具有先进大型语言模型(LLM)和计算机视觉技术的AR秘书代理。 该系统可以谨慎地向佩戴者提供实时信息,确定他们正在与谁交谈并总结以前的讨论。 为了验证AR秘书,我们与13名参与者进行了一项用户研究,并表明我们的技术可以有效地帮助用户在我们的研究中增加多达20%的内存增强。
Interacting with a significant number of individuals on a daily basis is commonplace for many professionals, which can lead to challenges in recalling specific details: Who is this person? What did we talk about last time? The advant of augmented reality (AR) glasses, equipped with visual and auditory data capture capabilities, presents a solution. In our work, we implemented an AR Secretary Agent with advanced Large Language Models (LLMs) and Computer Vision technologies. This system could disc...