Modern Hopfield network

Modern Hopfield networks are generalizations of the classical Hopfield networks that break the linear scaling relationship between the number of input features and the number of stored memories. This is achieved by introducing stronger non-linearities leading to super-linear memory storage capacity as a function of the number of feature neurons. The network still requires a sufficient number of hidden neurons.
The key theoretical idea behind the modern Hopfield networks is to use an energy function and an update rule that is more sharply peaked around the stored memories in the space of neuron’s configurations compared to the classical Hopfield network.

Classical Hopfield networks

Hopfield networks are recurrent neural networks with dynamical trajectories converging to fixed point attractor states and described by an energy function. The state of each model neuron is defined by a time-dependent variable, which can be chosen to be either discrete or continuous. A complete model describes the mathematics of how the future state of activity of each neuron depends on the known present or previous activity of all the neurons.
In the original Hopfield model of associative memory, the variables were binary, and the dynamics were described by a one-at-a-time update of the state of the neurons. An energy function quadratic in the was defined, and the dynamics consisted of changing the activity of each single neuron only if doing so would lower the total energy of the system. This same idea was extended to the case of being a continuous variable representing the output of neuron, and being a monotonic function of an input current. The dynamics became expressed as a set of first-order differential equations for which the "energy" of the system always decreased. The energy in the continuous case has one term which is quadratic in the , and a second term which depends on the gain function. While having many desirable properties of associative memory, both of these classical systems suffer from a small memory storage capacity, which scales linearly with the number of input features.

Discrete variables

A simple example of the Modern Hopfield network can be written in terms of binary variables that represent the active and inactive state of the model neuron.In this formula the weights represent the matrix of memory vectors, and the function is a rapidly growing non-linear function. The update rule for individual neurons can be written in the following form which states that in order to calculate the updated state of the -th neuron the network compares two energies: the energy of the network with the -th neuron in the ON state and the energy of the network with the -th neuron in the OFF state, given the states of the remaining neuron. The updated state of the -th neuron selects the state that has the lowest of the two energies.
In the limiting case when the non-linear energy function is quadratic these equations reduce to the familiar energy function and the update rule for the classical binary Hopfield network.
The memory storage capacity of these networks can be calculated for random binary patterns. For the power energy function the maximal number of memories that can be stored and retrieved from this network without errors is given byFor an exponential energy function the memory storage capacity is exponential in the number of feature neurons

Continuous variables

Modern Hopfield networks or Dense Associative Memories can be best understood in continuous variables and continuous time. Consider the network architecture, shown in Fig.1, and the equations for the neurons' state evolutionwhere the currents of the feature neurons are denoted by, and the currents of the memory neurons are denoted by . There are no synaptic connections among the feature neurons or the memory neurons. A matrix denotes the strength of synapses from a feature neuron to the memory neuron. The synapses are assumed to be symmetric, so that the same value characterizes a different physical synapse from the memory neuron to the feature neuron. The outputs of the memory neurons and the feature neurons are denoted by and, which are non-linear functions of the corresponding currents. In general these outputs can depend on the currents of all the neurons in that layer so that and. It is convenient to define these activation function as derivatives of the Lagrangian functions for the two groups of neuronsThis way the specific form of the equations for neuron's states is completely defined once the Lagrangian functions are specified. Finally, the time constants for the two groups of neurons are denoted by and, is the input current to the network that can be driven by the presented data.
General systems of non-linear differential equations can have many complicated behaviors that can depend on the choice of the non-linearities and the initial conditions. For Hopfield networks, however, this is not the case - the dynamical trajectories always converge to a fixed point attractor state. This property is achieved because these equations are specifically engineered so that they have an underlying energy function The terms grouped into square brackets represent a Legendre transform of the Lagrangian function with respect to the states of the neurons. If the Hessian matrices of the Lagrangian functions are positive semi-definite, the energy function is guaranteed to decrease on the dynamical trajectory This property makes it possible to prove that the system of dynamical equations describing temporal evolution of neurons' activities will eventually reach a fixed point attractor state.
In certain situations one can assume that the dynamics of hidden neurons equilibrates at a much faster time scale compared to the feature neurons,. In this case the steady state solution of the second equation in the system can be used to express the currents of the hidden units through the outputs of the feature neurons. This makes it possible to reduce the general theory to an effective theory for feature neurons only. The resulting effective update rules and the energies for various common choices of the Lagrangian functions are shown in Fig.2. In the case of log-sum-exponential Lagrangian function the update rule for the states of the feature neurons is the attention mechanism commonly used in many modern AI systems.

Relationship to classical Hopfield network with continuous variables

Classical formulation of continuous Hopfield networks can be understood as a special limiting case of the Modern Hopfield networks with one hidden layer. Continuous Hopfield Networks for neurons with graded response are typically described by the dynamical equations and the energy function where, and is the inverse of the activation function. This model is a special limit of the class of models that is called models A, with the following choice of the Lagrangian functions that, according to the definition, leads to the activation functions If we integrate out the hidden neurons the system of equations reduces to the equations on the feature neurons with, and the general expression for the energy reduces to the effective energy While the first two terms in equation are the same as those in equation, the third terms look superficially different. In equation it is a Legendre transform of the Lagrangian for the feature neurons, while in the third term is an integral of the inverse activation function. Nevertheless, these two expressions are in fact equivalent, since the derivatives of a function and its Legendre transform are inverse functions of each other. The easiest way to see that these two terms are equal explicitly is to differentiate each one with respect to. The results of these differentiations for both expressions are equal to. Thus, the two expressions are equal up to an additive constant. This completes the proof that the classical Hopfield network with continuous states is a special limiting case of the modern Hopfield network with energy.

General formulation

Biological neural networks have a large degree of heterogeneity in terms of different cell types. This section describes a mathematical model of a fully connected Modern Hopfield network assuming the extreme degree of heterogeneity: every single neuron is different. Specifically, an energy function and the corresponding dynamical equations are described assuming that each neuron has its own activation function and kinetic time scale. The network is assumed to be fully connected, so that every neuron is connected to every other neuron using a symmetric matrix of weights, indices and enumerate different neurons in the network, see Fig.3. The easiest way to mathematically formulate this problem is to define the architecture through a Lagrangian function that depends on the activities of all the neurons in the network. The activation function for each neuron is defined as a partial derivative of the Lagrangian with respect to that neuron's activity From the biological perspective one can think about as an axonal output of the neuron. In the simplest case, when the Lagrangian is additive for different neurons, this definition results in the activation that is a non-linear function of that neuron's activity. For non-additive Lagrangians this activation function can depend on the activities of a group of neurons. For instance, it can contain contrastive or divisive normalization. The dynamical equations describing temporal evolution of a given neuron are given by This equation belongs to the class of models called firing rate models in neuroscience. Each neuron collects the axonal outputs from all the neurons, weights them with the synaptic coefficients and produces its own time-dependent activity. The temporal evolution has a time constant, which in general can be different for every neuron. This network has a global energy function where the first two terms represent the Legendre transform of the Lagrangian function with respect to the neurons' currents. The temporal derivative of this energy function can be computed on the dynamical trajectories leading to The last inequality sign holds provided that the matrix is positive semi-definite. If, in addition to this, the energy function is bounded from below the non-linear dynamical equations are guaranteed to converge to a fixed point attractor state. The advantage of formulating this network in terms of the Lagrangian functions is that it makes it possible to easily experiment with different choices of the activation functions and different architectural arrangements of neurons. For all those flexible choices the conditions of convergence are determined by the properties of the matrix and the existence of the lower bound on the energy function.