A short survey on almost orthogonal vectors in a few specific large dimensions
Rami Luisto
几乎正交向矢量的概念,即表观相似度接近0的矢量,涉及纯数学和编码理论的主题,以球形包装和球形代码为幌子。 近年来,人工智能中高级语言模型的兴起对这一概念产生了新的兴趣,因为模型似乎将某些概念存储为高维空间中的几乎正交方向。 在这项调查中,我们通过三种方法代表了关于几乎正交向的一些想法:(1)几乎正交的数学理论,(2)语言模型嵌入空间的一些观察,(3)通过模拟生成大型几乎正交向矢量。
The concept of almost orthogonal vectors, i.e. vectors whose cosine similarity is close to 0, relates to topics both in pure mathematics and in coding theory under the guises of spherical packing and spherical codes. In recent years the rise of advanced language models in AI has created new interest in this concept as the models seem to store certain concepts as almost orthogonal directions in high-dimensional spaces. In this survey we represent some ideas regarding almost orthogonal vectors thr...