DAoG: decayed adaptation over gradients for parameter-free step size control

As the scale of parameters in deep learning models continues to grow, the cost of training such models increases accordingly, posing increasingly significant challenges for stochastic optimization methods. A central issue in gradient-based optimization lies in the selection of the step size, whose appropriateness directly affects training efficiency and model performance. To address this issue, a series of parameter-free optimization methods that do not require manual tuning of the step size have been proposed in recent years. Among them, DoG and its improved variant DoWG are the most representative. Despite demonstrating strong performance across various tasks, DoG and DoWG still suffer from performance instability or slow convergence under certain model architectures or training conditions. This paper introduces Decayed Adaptation over Gradients (DAoG), a novel parameter-free optimization method that systematically addresses these limitations. Our key innovation lies in incorporating a principled step size decay mechanism for the first time within the parameter-free optimization framework, which substantially enhances both optimization stability and model generalization. Additionally, a parameter compression strategy is employed to reduce sensitivity to the initial step size. Theoretical analysis demonstrates that DAoG exhibits favorable convergence properties under L-smooth and G-Lipschitz conditions. Empirical studies across representative tasks in natural language processing and computer vision demonstrate that DAoG outperforms both DoG and DoWG in terms of convergence speed and generalization performance. Notably, it even rivals or surpasses Adam with cosine annealing in several challenging scenarios. These theoretical and experimental results suggest that DAoG effectively mitigates the overly conservative step size issue in DoG and the instability problem in DoWG, thereby advancing the development of parameter-free optimization methods in deep learning.