Large width limits of neural networks

Theoretical approaches based on a large width limit

• The Neural Network Gaussian Process corresponds to the infinite width limit of Bayesian neural networks, and to the distribution over functions realized by non-Bayesian neural networks after random initialization.
• The same underlying computations that are used to derive the NNGP kernel are also used in deep information propagation to characterize the propagation of information about gradients and inputs through a deep network. This characterization is used to predict how model trainability depends on architecture and initializations hyper-parameters.
• The Neural Tangent Kernel describes the evolution of neural network predictions during gradient descent training. In the infinite width limit the NTK usually becomes constant, often allowing closed form expressions for the function computed by a wide neural network throughout gradient descent training.
• The study of infinite width neural networks with a different initial weight scaling leads to qualitatively different nonlinear training dynamics than those described by the fixed neural tangent kernel.
• Catapult dynamics describe neural network training dynamics in the case that logits diverge to infinity as the layer width is taken to infinity, and describe qualitative properties of early training dynamics.