Subspace Inference in SDE-Net for Bayesian Deep Neural Networks

Bayesian inference was once a gold standard for uncertainty estimation with neural networks. However, Bayesian inference is inefficient in high-dimensional parameter spaces of Deep Neural Networks (DNNs). A novel Non-Bayesian Neural Stochastic Differential Equation (SDE-Net) model quantifies epistemic uncertainty of DNNs from the perspective of dynamical systems. In this paper, we propose a subspace inference procedure in Non-Bayesian SDE-Net method for uncertainty estimation from Bayesian perspectives. This procedure contains two steps, first of all, the low-dimensional parameter subspaces of SDE-Net model are constructed to generate the first principal components from Stochastic Gradient Descent (SGD) trajectories, which involve various high-performance models. Secondly, Variational Inference (VI) and Elliptical Slice Sampling (ESS) methods are implemented in the constructed subspace to explore in the full parameter spaces for uncertainty estimation. The experimental results of Bayesian average on the derived posterior in the generated principal subspaces of model parameters show that accurate and well-calibrated results can be obtained for regression and classification tasks.