5.23.4 Tensor Product Array Representation

Tensor Product Array Representation is the practice of storing and manipulating the components of a tensor product element as a multi-dimensional array — a direct generalization of vectors as one-dimensional arrays and matrices as two-dimensional arrays — together with the conventions of indexing, memory layout, and reshaping that make such arrays usable in computation. Where component representation concerns the mathematical fact that a tensor has well-defined coordinates relative to a basis, array representation concerns the practical, computational question of how those coordinates are laid out, addressed, and stored as actual data.

From Components to a Multi-Dimensional Array

The Array as a Container for Components

Given the component expansion t = Σ c_{i₁...iₙ} (e^{(1)}_{i₁} ⊗ ... ⊗ e^{(n)}_{iₙ}) of a tensor in an n-factor tensor product, the array representation is simply the collection of all coefficients c_{i₁...iₙ}, organized as an n-dimensional array with one axis (or "mode") per tensor factor, and axis lengths equal to the dimensions of the corresponding factor spaces.

Order, Shape, and Size

The number of axes of the array is called the order of the tensor (matching the number of factors in the tensor product), and the tuple of axis lengths is called its shape; the total number of stored entries is the product of the axis lengths, matching the dimension formula for the tensor product itself, dim(V₁ ⊗ ... ⊗ Vₙ) = dim(V₁) ··· dim(Vₙ).

Diagram of Array Representation by Order

Indexing and Memory Layout

Linearizing a Multi-Index into Storage

Physical computer memory is one-dimensional, so a multi-index (i₁, ..., iₙ) must be mapped to a single linear memory offset; row-major (C-style) layout increments the last index fastest, while column-major (Fortran-style) layout increments the first index fastest, and this choice affects performance but not the mathematical content of the array.

Strides

The stride along each axis records how many linear memory positions must be skipped to advance one step along that axis; strides are computed from the shape and the chosen layout convention, and together with the shape they fully determine how a multi-index maps to a memory offset.

Reshaping and Flattening

Reshaping Without Changing Data

Because the array's linear storage does not inherently carry axis boundaries, a tensor's array representation can be reshaped — reinterpreting the same underlying data with a different shape — provided the total number of entries is unchanged, corresponding mathematically to reinterpreting the same components relative to a different (typically flattened or regrouped) choice of basis indexing on the tensor product.

Flattening to a Matrix or Vector

A common operation, called matricization or unfolding, reshapes an order-n array into a matrix by grouping some axes into "rows" and the rest into "columns"; this operation corresponds to using the associativity of the tensor product to regroup V₁ ⊗ ... ⊗ Vₙ as (Vᵢ₁ ⊗ ... ⊗ Vᵢₖ) ⊗ (Vⱼ₁ ⊗ ... ⊗ Vⱼ_{n-k}) and reading off components relative to the resulting two-factor basis.

Array Operations Corresponding to Tensor Operations

Elementwise Operations

Addition of two tensors and scalar multiplication correspond to elementwise addition and scaling of their array representations, directly mirroring the vector space structure of the tensor product.

Contraction as an Array Sum

Contracting a tensor over a pair of indices — summing the array over one axis paired against another — corresponds to applying a natural pairing between a tensor factor and its dual, translating an abstract tensor operation into an explicit summation over an array axis.

Practical Considerations

Sparse versus Dense Array Representation

When most components of a tensor are zero, a dense array representation (storing every entry, including zeros) can be wasteful; sparse array representations, storing only nonzero components together with their multi-indices, are used instead, trading simplicity of indexing for reduced memory use, without altering the underlying mathematical tensor.

Array Representation Depends on a Fixed Basis

As with any component-based representation, the array depends entirely on the bases chosen for each tensor factor; changing any factor's basis requires transforming the array according to the corresponding change-of-basis rule, since the array by itself carries no information about which bases produced it.

Significance of Array Representation

The Computational Interface to Tensor Products

Array representation is the concrete data structure through which essentially all practical, numerical work with tensor products is carried out, translating the abstract vector space V₁ ⊗ ... ⊗ Vₙ into an object that can be stored, indexed, and operated on by ordinary computer programs.

Foundation for Numerical Tensor Methods

The conventions of shape, stride, reshaping, and sparse storage developed for array representation underlie numerical tensor libraries used throughout scientific computing and machine learning, where large tensor products are manipulated efficiently by operating directly on their array representations rather than through abstract algebraic reasoning.