Dimensionality Reduction and Projection
Dimensionality Reduction and Projection simplify complex data by reducing features, preserving key information for efficient signal analysis and pattern recognition.
Dimensionality Reduction and Projection is the construction of a new Behavioral Representation by mapping an existing representation into a lower-dimensional or otherwise reduced coordinate space under an explicit preservation objective, a fitted or fixed mapping state, an output schema, and a comparability rule. It is critical to recognize that terms such as selection, projection, compression, factorization, embedding, visualization, reconstruction, and feature extraction are not synonyms and should not be conflated. Dimensionality reduction typically alters coordinate identity: the newly constructed coordinates are usually distinct from the original ones. A lower-dimensional representation can preserve some relations—such as variance or neighborhood structure—while distorting or discarding others. Importantly, dimensionality reduction is not inherently denoising, interpretability enhancement, behavioral abstraction, or guaranteed quality improvement; these are distinct properties that may or may not accompany reduction depending on method and context.
Meaning and Boundaries of Dimensionality Reduction
Dimensionality reduction is a mapping from a declared source representation space into an output representation with fewer effective or explicit coordinates under a stated notion of dimensionality. Projection broadly denotes the mapping that constructs output coordinates, subspaces, factors, embeddings, or other reduced structures. Projections can be linear or nonlinear, deterministic or random, fitted from data or fixed, supervised by target variables or unsupervised, and sometimes meaningful only for the fitted sample. This variety means projection is a broad methodological family rather than a single operation.
The following key terms define the projection framework:
- Source Representation: The original behavioral representation with a declared coordinate system and schema.
- Projection Mapping: The mathematical or algorithmic function transforming source coordinates into projected coordinates.
- Projection State: Any data-dependent or randomized object needed to reproduce the mapping (e.g., fitted parameters, random seed).
- Projected Space: The output coordinate space defined by the projection mapping and its state.
- Projected Coordinate: Each individual coordinate in the projected space, defined by the mapping and its state.
- Projected Instance: A single instance transformed from source representation to projected coordinates.
- Inverse/Reconstruction Mapping: Optional mapping from projected coordinates back to an approximation in the source space.
- Visualization Embedding: A projection optimized primarily for graphical display, often lacking out-of-sample transform semantics.
One algorithm name does not uniquely identify a projected space: factors such as input schema, scaling, fitted state, randomness, output dimension, component ordering, supervision, and software version can change representation identity. Therefore, each projected representation should be explicitly specified by its full Projection Definition.
The governing distinction between selection and projection is fundamental. Selection retains a declared subset of existing coordinates or components with their original identities intact; projection constructs new coordinates by combining one or more source dimensions. Aggregation summarizes several source values under a declared summary rule, and reshape/flatten changes structural indexing without altering coordinate identity. Neither aggregation nor reshaping should be called projection unless a new coordinate mapping is explicitly defined.
| Operation | Creates New Coordinates? | Primary Information Change | Critical Boundary |
|---|---|---|---|
| Selection | No | Reduces coordinate set without mixing | Retains original coordinate identities |
| Aggregation | No | Summarizes multiple values into one | Original coordinates are combined, not remapped |
| Reshape/Flatten | No | Changes structural indexing only | No coordinate change, just reordering or flattening |
| Linear Projection | Yes | Constructs linear combinations | New coordinates from linear basis, coordinate identity altered |
| Nonlinear Projection | Yes | Constructs nonlinear mappings | Potentially complex coordinate transformations |
| Random Projection | Yes | Random linear mapping | Reproducibility depends on preserved random state |
| Visualization Embedding | Yes | Fitted sample optimized display | May not generalize or support out-of-sample transform |
Dimensionality, Projection State, and Output Identity
Dimensionality notions must be explicitly stated because they critically affect interpretation:
- Ambient/Input Dimension: The number of declared coordinates in the source representation.
- Output Dimension: The number of coordinates in the projected space.
- Algebraic Rank: The rank of the projection mapping or data matrix, reflecting linear independence.
- Intrinsic/Effective Dimension: The minimal number of variables required to capture the relevant structure; requires a definition relative to behavior or information content.
- Information Content: The capacity of the representation to retain behaviorally relevant distinctions.
A representation may have many stored coordinates but low-rank or low-dimensional structure; conversely, a small output dimension does not guarantee sufficient retained information for the behavioral property of interest. The dimensionality notion must be declared whenever reduction claims depend on it.
A generic fitted projection is defined exactly as:
Here, bold x_i is the source Representation Instance i under the declared input schema; R is the Projection Definition; Φ_R is its source-to-projected mapping; θ_R are explicit projection parameters such as output dimension or fixed metric choices; Ω_R is optional fitted or random projection state; and bold z_i is the projected Representation Instance. Compatibility of projected coordinates requires compatible R, input schema, parameters, and state.
Projection state includes any data-dependent or randomized object needed to reproduce the mapping, such as centering/scaling values, basis/loadings, singular vectors, discriminant directions, random matrices, kernel reference sets, landmarks, graph/neighborhood structures, codebooks, optimization solutions, or fitted manifold coordinates. It is critical to distinguish reusable mapping state—which enables application to new instances—from merely storing the resulting training embedding.
Projected coordinate identity and versioning matter: new coordinates can have ordered component IDs, factor IDs, axes, basis vectors, or embedding dimensions, but these IDs are meaningful only relative to a specific projection state. Independently fitted spaces can differ by sign, permutation, rotation, reflection, scale, nonlinear deformation, or entirely different local optima, even when output dimension and algorithm settings match.
| Concept | Meaning | Common Misinterpretation |
|---|---|---|
| Input Dimension | Number of declared source coordinates | Mistaking it for intrinsic or effective dimension |
| Output Dimension | Number of projected coordinates | Assuming it equals intrinsic dimension or information content |
| Matrix Rank | Rank of data or projection matrix | Confusing rank with behavioral relevance |
| Intrinsic/Effective Dimension | Minimal dimension capturing relevant structure | Treating it as universal or fixed without stated notion |
| Projected Coordinate | New coordinate in output space defined by projection and state | Confusing with retained original coordinates |
| Projection State | Data-dependent or random parameters defining the mapping | Ignoring state differences leads to incompatible comparisons |
| Projected-Space Version | Specific fitted instance of projection space | Overlooking sign flips, rotations, or permutations as trivial |
Linear, Affine, and Factor-Space Projections
A generic centered linear projection is:
Here, bold x_i is a source vector with declared coordinate order; bold μ is the fitted or fixed reference center in the same source space; bold W is the declared projection-basis/loadings matrix whose columns define output directions under this convention; superscript T denotes transpose; and bold z_i are the projected coordinates. Centering, scaling, basis orientation, and source-coordinate order are part of representation identity. Nonlinear projections do not necessarily admit this affine-linear form.
Conceptually, linear projections can be:
- Orthogonal: Projection basis vectors are mutually orthogonal under a declared inner product; this is a geometric property, not statistical independence or behavioral separability.
- Oblique: Basis vectors are not orthogonal; projections may be easier to interpret or fit but lose geometric orthogonality.
- Sparse: Basis vectors have many zeros or near-zeros, improving interpretability or variable selection, but sparsity alone does not guarantee that a component corresponds to a single behavioral construct.
- Constrained/Nonnegative: Additional constraints (e.g., nonnegativity) restrict admissible solutions and thus define distinct projection identities.
A canonical example is PCA-like variance projection: centering and a fitted basis construct orthogonal directions ordered by captured variance under the adopted scaling. Maximum variance is an optimization criterion, not a behavioral importance, validity, causality, or discriminative usefulness definition. Furthermore, scaling source coordinates materially affects the fitted directions.
Low-rank factorization and truncated-SVD-like methods expose dominant linear structure and construct reduced coordinates. Centering policy critically influences results, especially for sparse inputs. Distinguish low-rank approximation of a data matrix from the claim that underlying behavior intrinsically has that dimension.
Whitening is an optional scale adjustment applied to projected coordinates to decorrelate and equalize component variances under a fitted covariance model. Whitening discards relative variance scale information and creates a distinct projected representation from unwhitened coordinates, even when using the same underlying directions.
| Coordinate Construction | What Is Favored | State Dependency | Interpretive Boundary |
|---|---|---|---|
| Orthogonal Linear Projection | Geometric orthogonality under inner product | Depends on centering/scaling | Orthogonality is geometric, not behavioral independence |
| Oblique Projection | Relaxed geometric constraints | Depends on constraints | Non-orthogonality affects geometry and interpretability |
| Variance-Ordered/PCA-Like Space | Directions maximizing variance | Fitted basis | Variance reflects data spread, not behavioral importance |
| Truncated-SVD-Like Space | Low-rank approximation of data matrix | Fitted basis | Approximation quality ≠ intrinsic behavior dimension |
| Sparse/Constrained Projection | Interpretability, variable selection | Constraints, sparsity | Sparsity ≠ behavioral construct purity |
| Whitened Space | Decorrelated, equal variance components | Fitted covariance | Whitening changes representation, discards variance scale |
Preservation Objectives, Reconstruction, and Information Loss
Preservation objectives explicitly define what aspects of the source representation the reduced representation is intended to retain. Examples include variance, pairwise distances, local neighborhoods, global distances, class/task separation, reconstruction, covariance across blocks, temporal structure, graph structure, or topology-like neighborhood relations. No finite-dimensional reduction can be judged generically without stating which relations matter and over which data or support domain.
A generic linear reconstruction from projected coordinates is:
Here, bold z_i are projected coordinates; bold W is the declared reconstruction/projection basis under the stated orthonormal-compatible convention; bold μ is the source-space center; and bold x̂_i is the reconstructed approximation in the source coordinate space. This reconstruction form does not apply universally to oblique, nonlinear, noninvertible, or visualization-only mappings. Low reconstruction error does not establish behavioral-semantic adequacy; important behavioral distinctions may be lost despite low reconstruction error.
Distinguish among:
- Exact inverse: A perfect, lossless backward mapping.
- Approximate inverse/reconstruction: A best-fit backward mapping minimizing reconstruction error.
- Preimage estimate: An approximate inverse in nonlinear or kernel spaces.
- Decoder-based reconstruction: Learned nonlinear inverse mappings (e.g., autoencoders).
- No inverse mapping: Some projections, especially embeddings and visualization-only methods, lack a defined inverse.
A mapping can preserve targeted geometry without supporting accurate reconstruction, while excellent reconstruction can discard low-variance but behaviorally important distinctions. Reconstruction is one preservation test, not a universal fidelity criterion.
Distance and neighborhood preservation are important but distinct: a projection may approximately preserve global pairwise distances, preserve only local neighborhoods, preserve rank-neighbor relationships, or distort density and inter-cluster distances yet remain useful for visualization. The intended geometry and source metric must be declared before interpreting reduced-space distances.
Random projection is a mapping whose state includes a sampled projection matrix or equivalent random state. Its primary justification is approximate distance preservation under stated dimensionality and sample assumptions. To ensure reproducibility, the random seed or realized matrix must be preserved; regenerating a different matrix creates a different projected coordinate system despite identical statistical construction.
| Preservation Target | What Success Supports | What It Does Not Guarantee |
|---|---|---|
| Variance | Capturing major data spread | Behavioral importance or causal relevance |
| Reconstruction | Recovering source-space approximations | Behavioral semantic adequacy |
| Global Distance | Approximate preservation of all pairwise distances | Faithful local neighborhood structure |
| Local Neighborhood | Preservation of nearest neighbors | Global geometric or metric fidelity |
| Class/Target Separation | Discriminative power for known labels | Label-independent behavioral validity |
| Cross-Block Association | Retention of covariance across data blocks | Individual block interpretability |
| Visualization Structure | Clear low-dimensional display of structure | True metric distances or semantic clustering |
Nonlinear, Manifold, Kernel, and Visualization Embeddings
Nonlinear projection and embedding construct output coordinates through mappings that need not preserve linear combinations or admit one global basis. Methodological families include kernel-based, neighborhood-based, geodesic/manifold-like, spectral-like, and optimization-defined embeddings. These are best treated as families rather than a catalog of named algorithms.
Manifold assumptions propose that observations lie on or near a lower-dimensional structured subset of the ambient representation space. This is an assumption or model of geometry, not proof behavior intrinsically has a unique true manifold. Sampling density, noise, disconnected regions, boundary effects, distance metric, and neighborhood choice materially influence the recovered embedding.
Visualization-oriented embeddings optimize fitted coordinates primarily to expose selected low-dimensional structure of a fitted sample. Local neighborhoods, apparent clusters, inter-cluster gaps, axis orientation, global distances, density, and empty regions can be distorted by the objective and hyperparameters. A compelling two-dimensional visualization should not be interpreted as direct evidence of discrete behavioral classes, metric distances, trajectories, or latent mechanisms.
Distinguish reusable out-of-sample mappings—which apply a learned transform to new compatible instances—from fitted-sample embeddings that optimize coordinates solely for the training sample without a general transform. This distinction is part of the Projection Definition; not every embedding is inductive.
| Mapping Character | Out-of-Sample Requirement | Primary Preservation Goal | Interpretive Risk |
|---|---|---|---|
| Linear Reusable Projection | Supports frozen transform | Linear subspace structure | Assumes linearity, may lose nonlinear structure |
| Kernel/Reference-Dependent Projection | Requires kernel/support reference set | Nonlinear similarity structure | Sensitive to kernel and reference choice |
| Neighborhood/Manifold Mapping | Requires neighborhood graph or manifold estimate | Local neighborhood structure | Sensitive to sampling, metric, and noise |
| Random Projection | Fixed random matrix needed | Approximate distance preservation | Randomness may obscure interpretability |
| Supervised Projection | Requires label or target conditioning | Class/target separation | Leakage risk if supervision is not controlled |
| Visualization-Only/Fitted-Sample Embedding | No out-of-sample transform | Sample-specific low-dim structure | Non-generalizable, distorted distances |
Fit–Apply Semantics, Identifiability, and Comparability
The fit–apply semantics separate estimation of data-dependent projection state using an eligible fitting set (fit) from the application of the frozen state on compatible new instances (apply/transform). Recomputing centering, scaling, basis, neighborhoods, kernel reference sets, codebooks, or other projection states separately on each comparison group places them in different coordinate systems, invalidating direct coordinate-wise comparisons.
Projection ambiguities include:
- Sign flips: One-dimensional components can reverse sign without changing the spanned subspace.
- Permutation: Ordering of components may change without altering subspace.
- Rotation: Joint subspaces remain stable while individual basis vectors rotate within them.
- Reflection: Geometric reflection preserves subspace but changes coordinate signs.
- Subspace ambiguity: Near-equal eigen/singular values cause instability of individual basis vectors but stable subspaces.
- Nonlinear embeddings: Can rotate, reflect, or deform under refitting.
Comparability may require component matching, alignment, subspace comparison, or invariant summaries rather than naïve coordinate equality.
Degeneracy and numerical conditioning affect projection stability: repeated or nearly repeated components, strong collinearity, insufficient support, extreme scaling differences, sparse structure, outliers, and numerical precision can destabilize fitted directions or inverse mappings. Instability of individual coordinates may coexist with stability of the represented subspace; quality assessment should target the identifiable object.
Out-of-sample validity and extrapolation matter: a transform may be mathematically defined for new inputs but scientifically unreliable if those inputs lie far outside the fitting distribution, violate the source schema, introduce unseen categories, occupy unsupported manifold regions, or differ in acquisition/preprocessing. Transformable does not imply transportable.
| Difference | Are Raw Coordinates Comparable? | Appropriate Remedy or Interpretation |
|---|---|---|
| Different Center/Scale | No | Align centering/scaling before comparison |
| Different Basis | No | Subspace comparison or basis alignment required |
| Sign Flip | No | Adjust sign or use sign-invariant metrics |
| Component Permutation | No | Permutation matching or subspace comparison |
| Near-Degenerate Subspace | No | Compare subspaces jointly, not individual components |
| Different Random Matrix | No | Reproducibility requires fixed random state |
| Different Kernel/Reference Set | No | Fix kernel and reference state; no direct coordinate comparison |
| Independent Nonlinear Fit | No | Compare using invariant summaries or alignment; no direct equality |
Supervision, Leakage, Dimensionality Choice, and Evaluation
Projection methods vary by supervisory conditioning:
- Unsupervised: No use of labels or outcomes during fitting.
- Supervised: Projection depends on labels, outcomes, classes, or behavioral references.
- Weakly supervised: Partial or indirect use of target information.
- Reference-conditioned: Depend on group membership or external annotations.
When target information influences mapping, it becomes part of representation identity and restricts independent evaluation claims. A supervised projection can be scientifically legitimate yet inappropriate as label-independent evidence.
Leakage occurs when projection state is fitted on held-out/test participants, future sessions, future time points in causal tasks, downstream target labels, reference labels used for both projection and final evaluation, or global normalization including evaluation data. Leakage can happen even when the final predictive model is fitted correctly because the representation itself consumes forbidden information.
Dimensionality choice is a use-specific decision, not a universal optimum. Evidence for dimension includes reconstruction curves, variance spectra, distance/neighborhood preservation, held-out task performance, stability, intrinsic-dimension diagnostics, interpretability, computational constraints, and scientific minimum-information requirements. Choosing dimension after inspecting final evaluation outcomes introduces optimistic bias.
Evaluation of projected representations includes held-out transformation, reconstruction (where defined), neighborhood/distance preservation, downstream utility, component/subspace stability, random-seed sensitivity, fit-cohort sensitivity, perturbation robustness, scaling sensitivity, outlier sensitivity, transportability, and preservation of known synthetic structure. No single metric establishes universal projection quality.
Structured Inputs, Worked Comparison, and Provenance
Structured-input projection boundaries depend on data type and semantics:
- Feature vectors: Projected directly under compatible coordinate semantics.
- Matrices/tensors: Projected along declared modes or after semantics-preserving flattening.
- Temporal sequences: Reduced per element, over time, or via sequence-level mappings.
- Graphs: Mapped into vector spaces through additional representation mechanisms.
In all cases, explicitly state which original axes or relations are combined or removed. Do not treat flattening, temporal pooling, graph construction, or multimodal fusion as projection merely because they precede reduction.
Integrated worked comparison
Consider a frozen 40-coordinate Behavioral Feature-Space Representation built from heterogeneous descriptors. Various reduction scenarios illustrate projection nuance:
- Selection of 12 original coordinates: Retains coordinate identity, no new axes created.
- Projection of all 40 into an 8-dimensional centered PCA-like space: Orthogonal directions fitted on training participants only.
- Whitening that same space: Decorrelates and equalizes component variances, changing the representation.
- Applying the frozen projection to a new participant: Uses fixed projection state for out-of-sample transform.
- Reconstructing the new vector approximately: Linear reconstruction from projected coordinates.
- Sign-flipped component: Preserves the same one-dimensional subspace but reverses coordinate sign.
- Near-degenerate pair of components: Joint subspace stable; individual coordinates rotate.
- Random 8-dimensional projection under one frozen matrix: Random projection with fixed random state.
- Supervised 4-dimensional projection conditioned on reference labels: Incorporates target information, restricting evaluation validity.
- 2D visualization-oriented nonlinear embedding with no assumed ordinary transform: Optimized for fitted sample display, no out-of-sample mapping.
Demonstrated failures include:
- Leakage by fitting projection state on evaluation participants, contaminating results.
- Incompatible independent refit producing different projection space.
- Different output dimension choices yielding tradeoffs in preservation.
- Low reconstruction error failing to preserve scientifically important low-variance behavioral distinctions.
Dimensionality Reduction and Projection provenance includes information required to reproduce and scientifically interpret both the Projection Definition and each projected instance:
- Source Representation Definition/version and coordinate order
- Fitting population/support and eligibility rules
- Centering/scaling state
- Projection family and preservation objective
- Explicit parameters and output dimension/order
- Fitted basis/loadings or equivalent projection state
- Random seed/realized matrix
- Kernel/reference/neighborhood state
- Supervision/target provenance
- Inverse/reconstruction semantics
- Out-of-sample capability
- Component/subspace identifiability conventions
- Whitening, sparsity, structured-input mapping
- Missing-data handling policies
- Leakage constraints
- Dimensionality-selection rules
- Evaluation metrics
- Implementation/version details
- Uncertainty and sensitivity assessments
- Comparability requirements
A defensible reduced representation states which source distinctions were combined or removed, what relation the mapping attempts to preserve, what state defines the projected coordinate system, and which comparisons remain scientifically valid.