5.23.5 Tensor Product Representation Limit

Tensor Product Representation Limit is the collective name for the circumstances under which the ordinary component or array representation of a tensor product element becomes impractical, ambiguous, or altogether unavailable, including the exponential growth of storage size with tensor order, the impossibility of a finite component array for infinite-dimensional factor spaces, and the inherent basis-dependence that limits how directly components can be compared or interpreted. Understanding these limits is necessary to know when the component representation, however useful computationally, ceases to be a faithful or feasible stand-in for the abstract tensor it represents.

The Curse of Dimensionality

Exponential Growth in Storage Size

For an n-factor tensor product of spaces each of dimension d, the component array has dⁿ entries, since the dimension of V₁ ⊗ ... ⊗ Vₙ is the product of the individual dimensions. Even modest values of d and n produce array sizes that grow far faster than either quantity alone, a phenomenon commonly called the curse of dimensionality.

Practical Consequence for High-Order Tensors

means that representing, say, a tensor of order 10 with each factor of dimension 100 would require 100^{10} = 10^{20} components, an amount of data far beyond what can be stored or manipulated directly, forcing practitioners toward structured or compressed representations (such as low-rank decompositions) rather than full dense arrays whenever the order or dimension is large.

Diagram of Storage Growth

Failure of Finite Component Representation in Infinite Dimensions

No Finite Array When a Factor Is Infinite-Dimensional

If any factor space Vᵢ is infinite-dimensional, its basis is an infinite index set, so the component array of a general element would need infinitely many entries along that axis; although any individual tensor still has only finitely many nonzero components (since it is a finite sum of simple tensors), there is no finite bound on which basis indices might be involved across different tensors, and no single finite array shape can accommodate every element of the space.

Sparse, Basis-Dependent Description as a Partial Remedy

In the infinite-dimensional case, tensors are typically described not by a full array but by listing only their finitely many nonzero components together with their (from an infinite index set) basis indices, which remains a valid and unique description by the basis expansion theorem, but abandons the fixed-shape array format available in the finite-dimensional case.

Basis-Dependence as an Interpretive Limit

Components Alone Do Not Determine Geometric or Algebraic Meaning

Because a component array depends entirely on the bases chosen for each factor, comparing two tensors' component arrays directly (for instance, checking whether they look "similar") is meaningful only if both were expressed relative to the same, fixed bases; without accounting for this, the same abstract tensor can appear to have arbitrarily different-looking arrays.

Rank Is Robust, Raw Components Are Not

While the tensor's rank and the tensor itself as an abstract object are basis-independent, most other properties one might read off an array by inspection (such as which entries happen to be zero, or the specific numerical values present) are basis-dependent and can be radically altered by a change of basis, limiting how much can be inferred from an array representation without also knowing the basis it was computed against.

Numerical Limits in Practice

Loss of Precision in Large Arrays

When components are stored as floating-point numbers, operations such as tensor products of maps or contractions accumulate rounding error across the (potentially very large) array, and this numerical imprecision compounds more severely as array size grows, a practical representation limit distinct from, but related to, the purely combinatorial curse of dimensionality.

Rank Is Not Always Computable Efficiently

Even when a component array is available in full, determining the exact tensor rank of a general higher-order tensor (order three or more) is computationally difficult in general, unlike the matrix case (order two), where rank is efficiently computable; this is a representation limit specific to the interpretation, rather than the storage, of tensor components.

Significance of Representation Limits

Motivating Structured and Approximate Representations

Recognizing these representation limits motivates the widespread use of structured tensor formats — low-rank decompositions, tensor train formats, and sparse representations — that approximate or exactly capture a tensor using far less data than a full dense component array, trading exact and complete representation for computational feasibility.

Clarifying the Boundary Between Abstract Theory and Computation

Representation limits mark the boundary at which the abstract, basis-free theory of the tensor product — where dimension, associativity, and symmetry hold without qualification — must be supplemented by additional computational techniques whenever an explicit, finite, basis-dependent array is required but not directly feasible to construct or manipulate in full.