Where You Place the Norm Matters: From Prejudiced to Neutral Initializations
Emanuele Francazi, Francesco Pinto, Aurelien Lucchi, Marco Baity-Jesi
规范化层,如批次规范化和层标准化,是现代神经网络的核心组成部分,被广泛采用以提高训练稳定性和概括性。 虽然它们的实际有效性有据可查,但从初始化开始,对规范化如何影响模型行为的详细理论理解仍然是一个重要的开放问题。 在这项工作中,我们研究在隐藏层中规范化的存在和放置如何影响训练开始之前网络预测的统计属性。 特别是,我们研究这些选择如何在初始化时塑造类预测的分布,从无偏见(中立)到高度集中(预判)到类的一个子集。 我们的分析表明,归一化放置会诱发神经网络初始预测行为的系统差异,进而塑造学习的动态。 通过将架构选择与初始化时的预测统计联系起来,我们的工作提供了对规范化如何影响早期训练行为的原则性理解,并为更可控和可解释的网络设计提供指导。
Normalization layers, such as Batch Normalization and Layer Normalization, are central components in modern neural networks, widely adopted to improve training stability and generalization. While their practical effectiveness is well documented, a detailed theoretical understanding of how normalization affects model behavior, starting from initialization, remains an important open question. In this work, we investigate how both the presence and placement of normalization within hidden layers inf...