Hilbert projection theorem

In mathematics, the Hilbert projection theorem is a famous result of convex analysis that says that for every vector in a Hilbert space and every nonempty closed convex there exists a unique vector for which is minimized over the vectors ; that is, such that for every

Finite dimensional case

Some intuition for the theorem can be obtained by considering the first order condition of the optimization problem.
Consider a finite dimensional real Hilbert space with a subspace and a point If is a or of the function defined by, then derivative must be zero at
In matrix derivative notation:
Since is a vector in that represents an arbitrary tangent direction, it follows that must be orthogonal to every vector in

Statement

Proof by reduction to a special case

It suffices to prove the theorem in the case of because the general case follows from the statement below by replacing with

Consequences

If then
which implies
Let where is the underlying scalar field of and define
which is continuous and linear because this is true of each of its coordinates
The set is closed in because is closed in and is continuous.
The kernel of any linear map is a vector subspace of its domain, which is why is a vector subspace of
Let
The Hilbert projection theorem guarantees the existence of a unique such that .
Let so that and it remains to show that
The inequality above can be rewritten as:
Because and is a vector space, and which implies that
The previous inequality thus becomes
or equivalently,
But this last statement is true if and only if every Thus

Properties

Expression as a global minimum
The statement and conclusion of the Hilbert projection theorem can be expressed in terms of global minimums of the following functions. Their notation will also be used to simplify certain statements.
Given a non-empty subset and some define a function
A of if one exists, is any point in such that
in which case is equal to the of the function which is:
Effects of translations and scalings
When this global minimum point exists and is unique then denote it by explicitly, the defining properties of are:
The Hilbert projection theorem guarantees that this unique minimum point exists whenever is a non-empty closed and convex subset of a Hilbert space.
However, such a minimum point can also exist in non-convex or non-closed subsets as well; for instance, just as long is is non-empty, if then
If is a non-empty subset, is any scalar, and are any vectors then
which implies:
Examples
The following counter-example demonstrates a continuous linear isomorphism for which
Endow with the dot product, let and for every real let be the line of slope through the origin, where it is readily verified that
Pick a real number and define by .
Then is an invertible continuous linear operator that satisfies and
so that and
Consequently, if with and if then

Iterated projections

For any closed convex nonempty subset, let be the projection function.
If there are multiple closed convex subsets, then one can approximate the projection operator by applying in sequence, then do it again and again. That is, one can approximate as. The Kaczmarz method is a commonly used special case. Such methods can be computationally effective. For example, if is a complicated shape, then projecting directly to may be difficult. However, can be approximated as an intersection of simple objects like half-spaces, hyperplanes, finite-dimensional subspaces, or cones.
If is a closed subspace, then it is convex. In this case, the projection function is an orthogonal projection. A classic theorem states that, if are closed subspaces, then