Humble your Overconfident Networks: Unlearning Overfitting via Sequential Monte Carlo Tempered Deep Ensembles
Andrew Millard, Zheng Zhao, Joshua Murphy, Simon Maskell
顺序蒙特卡洛(SMC)方法为贝叶斯不确定性量化提供了一种有原则的方法,但传统上受到全批梯度评估需求的限制。 我们通过将随机梯度哈密尔顿蒙特卡洛(SGHMC)提案纳入SMC,引入可扩展变体,从而实现基于小型批量的高效采样。 我们生成的 SRCSGHMC 算法优于标准随机梯度下降 (SGD) 和深度集成,包括图像分类、异种 (OOD) 检测和传输学习任务。 我们进一步证明,SMCSGHMC可减轻过拟合并改善校准,为将预训练的神经网络转换为校准良好的贝叶斯模型提供了灵活、可扩展的途径。
Sequential Monte Carlo (SMC) methods offer a principled approach to Bayesian uncertainty quantification but are traditionally limited by the need for full-batch gradient evaluations. We introduce a scalable variant by incorporating Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) proposals into SMC, enabling efficient mini-batch based sampling. Our resulting SMCSGHMC algorithm outperforms standard stochastic gradient descent (SGD) and deep ensembles across image classification, out-of-distrib...