Iterated Population Based Training with Task-Agnostic Restarts
Alexander Chebykin, Tanja Alderliesten, Peter A. N. Bosman
超参数优化(HPO)可以解除神经网络调整超参数(HP)的负担。 基于人口培训(PBT)家族的HPO算法通过在权重优化的几步中动态调整HPs而高效。 最近的结果表明,HP更新之间的步骤数量是所有PBT变体的重要元HP,可以对其性能产生重大影响。 然而,没有任何方法或直觉可用于有效地确定其价值。 我们引入了基于人口的培训(IPBT),这是一种新颖的PBT变体,通过重新启动自动调整此HP,以任务无关的方式重用权重信息,并利用时间变化的贝叶斯优化重新初始化HP。 对8个图像分类和强化学习任务的评估表明,平均而言,我们的算法匹配或优于5个以前的PBT变体和其他HPO算法(随机搜索,ASHA,SMAC3),而不需要预算增加或其HP的任何变化。 源代码可在https://github.com/AwesomeLemon/IPBT查阅。
Hyperparameter Optimization (HPO) can lift the burden of tuning hyperparameters (HPs) of neural networks. HPO algorithms from the Population Based Training (PBT) family are efficient thanks to dynamically adjusting HPs every few steps of the weight optimization. Recent results indicate that the number of steps between HP updates is an important meta-HP of all PBT variants that can substantially affect their performance. Yet, no method or intuition is available for efficiently setting its value. ...
