Tensor (machine learning)
In machine learning, the term tensor informally refers to two different concepts a way of organizing data and a multilinear transformation. Data may be organized in a multidimensional array, informally referred to as a "data tensor"; however, in the strict mathematical sense, a tensor is a multilinear mapping over a set of domain vector spaces to a range vector space. Observations, such as images, movies, volumes, sounds, and relationships among words and concepts, stored in an M-way array, may be analyzed either by artificial neural networks or tensor methods.
Tensor decomposition factors data tensors into smaller tensors. Operations on data tensors can be expressed in terms of matrix multiplication and the Kronecker product. The computation of gradients, a crucial aspect of backpropagation, can be performed using software libraries such as PyTorch and TensorFlow.
Computations are often performed on graphics processing units using CUDA, and on dedicated hardware such as Google's Tensor Processing Unit or Nvidia's Tensor core. These developments have greatly accelerated neural network architectures, and increased the size and complexity of models that can be trained.
History
A tensor is by definition a multilinear map. In mathematics, this may express a multilinear relationship between sets of algebraic objects. In physics, tensor fields, considered as tensors at each point in space, are useful in expressing mechanics such as stress or elasticity. In machine learning, the exact use of tensors depends on the statistical approach being used.In 2001, the field of signal processing and statistics were making use of tensor methods. Pierre Comon surveys the early adoption of tensor methods in the fields of telecommunications, radio surveillance, chemometrics and sensor processing. Linear tensor rank methods analyzed M-way arrays composed of higher order statistics that were employed in blind source separation problems to compute a linear model of the data. He noted several early limitations in determining the tensor rank and efficient tensor rank decomposition.
In the early 2000s, multilinear tensor methods crossed over into computer vision, computer graphics and machine learning with papers by Vasilescu or in collaboration with Terzopoulos, such as Human Motion Signatures, TensorFaces TensorTextures and Multilinear Projection. Multilinear algebra, the algebra of higher-order tensors, is a suitable and transparent framework for analyzing the multifactor structure of an ensemble of observations and for addressing the difficult problem of disentangling the causal factors based on second order or higher order statistics associated with each causal factor.
Tensor factor analysis disentangles and reduces the influence of different causal factors with multilinear subspace learning.
When treating an image or a video as a 2- or 3-way array, i.e., "data matrix/tensor", tensor methods reduce spatial or time redundancies as demonstrated by Wang and Ahuja.
Yoshua Bengio,
Geoff Hinton
and their collaborators briefly discuss the relationship between deep neural networks and tensor factor analysis beyond the use of M-way arrays as inputs. One of the early uses of tensors for neural networks appeared in natural language processing. A single word can be expressed as a vector via Word2vec. Thus a relationship between two words can be encoded in a matrix. However, for more complex relationships such as subject-object-verb, it is necessary to build higher-dimensional networks. In 2009, the work of Sutskever introduced Bayesian Clustered Tensor Factorization to model relational concepts while reducing the parameter space. From 2014 to 2015, tensor methods become more common in convolutional neural networks. Tensor methods organize neural network weights in a "data tensor", analyze and reduce the number of neural network weights. Lebedev et al. accelerated CNN networks for character classification by using 4D kernel tensors.
Definition
Let be a field such as the real numbers or the complex numbers. A tensor is a multilinear transformation from a set of domain vector spaces to a range vector space:Here, and are positive integers, and is the number of modes of a tensor. The dimensionality of mode is, for .
In statistics and machine learning, an image is vectorized when viewed as a single observation, and a collection of vectorized images is organized as a "data tensor". For example, a set of facial images with pixels that are the consequences of multiple causal factors, such as a facial geometry, an expression, an illumination condition, and a viewing condition may be organized into a data tensor where are the total number of facial geometries, are the total number of expressions, are the total number of illumination conditions, and are the total number of viewing conditions. Tensor factorizations methods such as TensorFaces and multilinear independent component analysis factorizes the data tensor into a set of vector spaces that span the causal factor representations, where an image is the result of tensor transformation that maps a set of causal factor representations to the pixel space.
Another approach to using tensors in machine learning is to embed various data types directly. For example, a grayscale image, commonly represented as a discrete 2-way array with dimensionality where are the number of rows and are the number of columns. When an image is treated as 2-way array or 2nd order tensor, tensor factorization methods compute the image column space, the image row space and the normalized PCA coefficients or the ICA coefficients.
Similarly, a color image with RGB channels, may be viewed as a 3rd order data tensor or 3-way array.--------
In natural language processing, a word might be expressed as a vector via the Word2vec algorithm. Thus becomes a mode-1 tensor
The embedding of subject-object-verb semantics requires embedding relationships among three words. Because a word is itself a vector, subject-object-verb semantics could be expressed using mode-3 tensors
In practice the neural network designer is primarily concerned with the specification of embeddings, the connection of tensor layers, and the operations performed on them in a network. Modern machine learning frameworks manage the optimization, tensor factorization and backpropagation automatically.
As unit values
Tensors may be used as the unit values of neural networks which extend the concept of scalar, vector and matrix values to multiple dimensions.The output value of single layer unit is the sum-product of its input units and the connection weights filtered through the activation function :
where
If each output element of is a scalar, then we have the classical definition of an artificial neural network. By replacing each unit component with a tensor, the network is able to express higher dimensional data such as images or videos:
This use of tensors to replace unit values is common in convolutional neural networks where each unit might be an image processed through multiple layers. By embedding the data in tensors such network structures enable learning of complex data types.
In fully connected layers
Tensors may also be used to compute the layers of a fully connected neural network, where the tensor is applied to the entire layer instead of individual unit values.The output value of single layer unit is the sum-product of its input units and the connection weights filtered through the activation function :
The vectors and of output values can be expressed as a mode-1 tensors, while the hidden weights can be expressed as a mode-2 tensor. In this example the unit values are scalars while the tensor takes on the dimensions of the network layers:
In this notation, the output values can be computed as a tensor product of the input and weight tensors:
which computes the sum-product as a tensor multiplication.
This formulation of tensors enables the entire layer of a fully connected network to be efficiently computed by mapping the units and weights to tensors.
In convolutional layers
A different reformulation of neural networks allows tensors to express the convolution layers of a neural network. A convolutional layer has multiple inputs, each of which is a spatial structure such as an image or volume. The inputs are convolved by filtering before being passed to the next layer. A typical use is to perform feature detection or isolation in image recognition.Convolution is often computed as the multiplication of an input signal with a filter kernel. In two dimensions the discrete, finite form is:
where is the width of the kernel.
This definition can be rephrased as a matrix-vector product in terms of tensors that express the kernel, data and inverse transform of the kernel.
where and are the inverse transform, data and kernel. The derivation is more complex when the filtering kernel also includes a non-linear activation function such as sigmoid or ReLU.
The hidden weights of the convolution layer are the parameters to the filter. These can be reduced with a pooling layer which reduces the resolution of the data, and can also be expressed as a tensor operation.
Tensor factorization
An important contribution of tensors in machine learning is the ability to factorize tensors to decompose data into constituent factors or reduce the learned parameters. Data tensor modeling techniques stem from the linear tensor decomposition and the multilinear tensor decompositions.Tucker decomposition
, for example, takes a 3-way arrayand decomposes the tensor into three matrices and a smaller tensor. The shape of the matrices and new tensor are such that the total number of elements is reduced. The new tensors have shapes
Then the original tensor can be expressed as the tensor product of these four tensors:
In the example shown in the figure, the dimensions of the tensors are
The total number of elements in the Tucker factorization is
The number of elements in the original is 144, resulting in a data reduction from 144 down to 110 elements, a reduction of 23% in parameters or data size. For much larger initial tensors, and depending on the rank of the tensor, the gains can be more significant.
The work of Rabanser et al. provides an introduction to tensors with more details on the extension of Tucker decomposition to N-dimensions beyond the mode-3 example given here.