Behavioral Representation

Behavioral Representation models human actions through signals, linking observable behavior to cognitive processes in signal processing systems.

Behavioral Representation is the explicit organization or encoding of identified behavioral-signal evidence into a structured form whose elements, axes, relations, ordering, mapping, and information semantics are declared sufficiently for scientific interpretation and subsequent analysis. It is crucial to establish that a Behavioral Representation is not the raw behavioral phenomenon itself, nor is it automatically the source signal. It is not synonymous with a Behavioral Signal Descriptor, feature, feature set, embedding, latent state, annotation, behavioral reference, or model output. Furthermore, Behavioral Representation is not merely defined by being stored as a vector, matrix, tensor, sequence, graph, or any other computational container. Instead, it is a scientifically declared construction that organizes behavioral evidence into an interpretable and analyzable structure.

Meaning and Boundaries of Behavioral Representation

A Behavioral Representation is a structured encoding of identified behavioral evidence produced under a declared representation mapping, source/support semantics, parameterization, schema, and fitted or learned state when applicable. Representation concerns how evidence is organized and made available as a scientific object; it can preserve, expose, suppress, discretize, reorder, relate, compress, or transform information without changing the historical fact of what source evidence was originally observed.

Distinct concepts must be clearly distinguished:

Behavioral Evidence: The raw or processed data or observations identified as relevant behavioral phenomena.
Signal Transformation: An operation that produces another signal-like or coefficient object from original evidence, such as filtering or time-frequency decomposition.
Descriptor: A characterization of a declared property or aspect of the evidence, often a measurable or computed attribute.
Feature: A quantity used in a particular analytical task, typically selected or engineered for specific inference or classification goals.
Representation: An organization of evidence into a declared, structured form with specified semantics and mapping.
Annotation / Behavioral Reference: Objects carrying assignment or evidential semantics, such as labels, event tags, or reference markers.
Model Output: An inference, prediction, or estimation result derived from applying a model to input data.

One object can participate in constructing another without their scientific identities becoming synonymous. For example, a descriptor may be part of a representation, but the descriptor itself does not constitute the full representation.

Representation is not an intrinsic object hidden inside data. The same behavioral evidence can admit several scientifically legitimate representations because different mappings preserve different distinctions, impose different structures, and support different operations. Conversely, numerically similar representation values can have different meaning when their source evidence, schema, axes, component identities, fitted state, or mapping differ.

Object	Primary Scientific Role	Identity Comes From	Critical Non-Equivalence
Behavioral Evidence	Raw or identified behavioral data	Source observation or measurement	Not a constructed or transformed object
Signal Transformation	Produces alternate signal-like or coefficient data	Transformation operation applied to evidence	Not a representation or feature, but a transformed signal
Descriptor	Characterizes declared property	Property definition and extraction method	Different from representation or feature
Feature	Analytical quantity for specific task	Task context and selection criteria	Not equivalent to representation or raw evidence
Representation	Structured organization of evidence	Declared mapping, schema, source/support, state	Not raw data, feature set, annotation, or output
Annotation/Reference	Carries assignment or evidential semantics	Labeling or referencing protocol	Different from representation or feature
Model Output	Inference or prediction result	Model and input data	Not an evidence encoding; result of a model

Evidence, Mapping, and Representation Identity

Behavioral evidence that can legitimately be represented includes but is not limited to:

Prepared signal samples or channels,
Descriptor instances and contours,
Events,
Trajectories and poses,
Spectral or time-frequency objects,
Symbolic or linguistic evidence,
Relational objects,
Scientifically compatible collections of any of the above.

The representational semantics inherit whether the source is direct observation, transformed evidence, a derived characterization, an inferred object, or a constructed reference. This evidential distinction remains critical in interpreting the representation.

A generic representation mapping can be expressed as:

Z_{i} = Φ_{R} (E_{i} | S_{i}; θ_{R}, Ω_{R})

Here,

i indexes the representation instance,
R is the declared representation definition,
E_i is the identified behavioral evidence supplied to instance i,
S_i is its declared support or evidence scope,
Φ_R is the representation mapping defined by R,
θ_R are the resolved explicit representation parameters,
Ω_R is optional fitted or learned state required by the mapping,
Z_i is the resulting Representation Instance.

Z_i can be vectorial, sequential, symbolic, geometric, relational, graph-structured, latent, or another explicitly specified form rather than necessarily a point in an ordinary Euclidean vector space.

It is essential to distinguish four concepts:

Representation Definition: The reusable specification establishing semantics, schema, parameterization, and construction rules.
Representation Mapping: The operational rule or function that realizes the definition on given evidence, parameters, and fitted state.
Representation Instance: The binding of a representation definition to identified evidence, declared support, resolved parameters, and fitted/learned state, producing a concrete encoded content.
Represented Value or Structure: The concrete encoded content or data structure output by the instance.

Reusing one representation definition on another participant, session, episode, or support normally creates another instance rather than a new scientific definition.

Explicit parameters include coordinate ordering, support size, normalization policy, vocabulary choice, graph construction rule, or output dimension. These are declared directly.

Fitted or learned state encompasses means or scales estimated from training data, projection bases, dictionaries, codebooks, clustering states, encoder weights, checkpoints, or other data-dependent objects. Changing fitted state can change the representation even when the algorithm name and output shape remain unchanged.

Representation Schema, Structure, and Space

A Representation Schema is the declared organization needed to interpret a representation's structure. It includes:

Form (vector, sequence, graph, etc.),
Axes and their meaning,
Shape or variable-shape policies,
Component or token identity,
Ordering and coordinates,
Units or value semantics,
Masks and missingness states,
Sparsity,
Relation identity,
Validity flags.

Identical numerical arrays can represent scientifically different objects when these schema semantics differ.

A Representation Space is the set or structured domain in which Representation Instances are interpreted together with any meaningful geometry, topology, ordering, metric, similarity, algebraic operation, or structural constraint. Euclidean distance, component-wise arithmetic, averaging, interpolation, or nearest-neighbor reasoning is meaningful only when the representation space supports those operations under the declared semantics.

Representation forms include fixed-shape, variable-length or ragged, dense, sparse, set-valued, sequence-valued, matrix/tensor, symbolic, and graph-structured. Storage convenience should never dictate scientific form: padding a variable-length sequence, flattening a tensor, ordering a set, or serializing a graph can change the computational container without changing—or can accidentally change—the intended representation semantics.

Representation Form	Structure Preserved	Meaningful Operations	Risk If Treated as a Generic Array
Vector	Fixed coordinate identity and order	Component-wise arithmetic, averaging	Loss of component semantics if components are mixed
Sequence	Ordering, temporal or ordinal index	Temporal alignment, interpolation, masking	Ignoring order loses temporal meaning
Matrix/Tensor	Multi-axis structure and relations	Tensor algebra, slicing, reshaping	Flattening loses multi-dimensional relations
Set	Unordered elements, membership	Set operations, membership tests	Imposing order introduces artificial structure
Sparse Structure	Nonzero pattern, sparsity pattern	Sparse arithmetic, efficient storage	Treating as dense wastes resources or masks sparsity
Symbolic/Token Structure	Token identity, vocabulary, order	Symbolic parsing, token substitution	Numeric IDs treated as metric distort symbolic meaning
Graph/Relational Structure	Nodes, edges, direction, weights	Graph traversal, connectivity, relation inference	Flattening destroys graph topology and relations
Latent Space	Learned latent coordinates	Similarity, clustering, embedding operations	Coordinates may lack stable semantic interpretation

Principal Forms of Behavioral Representation

Behavioral Representation forms are overlapping scientific organizations rather than mutually exclusive container classes. A representation can simultaneously be temporal and geometric, sparse and symbolic, graph-structured and learned, or feature-space and sequence-valued. Classification should follow the structure and information semantics that matter to the scientific question rather than forcing every representation into one exclusive label.

Form	Primary Structure Preserved	Representative Input Evidence	Characteristic Information Risk
Feature-Space	Explicit coordinate/component identity	Descriptors, transformed quantities	Component semantics lost if treated as unstructured vector
Temporal Sequence	Ordered elements indexed by time/frame	Time-series, event sequences	Losing order loses temporal or causal context
Discrete/Symbolic	Token identity, vocabulary, order	Symbol codes, linguistic tokens	Numeric IDs imply metric relations that are not behaviorally meaningful
Spatial/Geometric	Coordinates, landmarks, poses, frames	Trajectories, poses, spatial landmarks	Flattening loses spatial frame and geometric relations
Relational/Graph	Entities and relations (edges, weights)	Social networks, interaction graphs	Flattening destroys topology and relation semantics
Latent/Learned	Data-dependent learned coordinates	Encoded embeddings, latent factors	Coordinates may lack stable or interpretable semantics

Feature-space representations are organized collections of explicitly identified coordinates or components used to place comparable instances in a common representational space. Coordinates can originate from descriptors, transformed quantities, measurements, encoded categories, or other declared sources. A feature-space representation is more than a list of numbers because component identity, order, units/scaling, missingness, and comparability determine its meaning.

Temporal sequence representations are ordered elements indexed by time, frame, event order, support anchor, or another declared sequence coordinate. They preserve ordering, spacing or timestamps when scientifically meaningful, variable-length status, masks, and causal or future-inclusive evidence semantics. A sequence representation records ordered evidence; it does not by itself model temporal dependence, state dynamics, transitions, or forecasting laws.

Discrete and symbolic representations are encodings over a declared finite or countable vocabulary, alphabet, token system, codebook, state set, event type set, or structured symbolic grammar. They preserve symbol identity, ordering, unknown/missing states, vocabulary version, and whether symbols originate from direct coding, quantization, event extraction, linguistic analysis, clustering, or another procedure. Numerical token IDs do not create metric behavioral meaning.

Spatial and geometric representations are structured encodings whose coordinate frames, landmarks, poses, spatial relations, trajectories, meshes, regions, or geometric constraints are part of the represented information. They preserve dimensionality, frame, units, entity identity, calibration, ordering, and projection state. Flattening geometric coordinates into a vector does not erase their geometric semantics.

Relational and graph representations are encodings in which entities and their declared relationships are explicit structural elements. They preserve node/entity identity, edge/relation semantics, directionality, weights, temporal status where applicable, and graph construction policy. Graph topology can encode observed, derived, thresholded, or learned relations; the presence of an edge therefore does not automatically mean physical interaction, causal influence, or social connection.

Latent and learned representations are structured outputs whose coordinates or relations are substantially determined by fitted data-dependent mappings such as learned encoders, factors, dictionaries, codebooks, or other estimated states. Learned representations can be vectors, sequences, tensors, graphs, or other forms. Latent coordinates need not have stable coordinate-wise behavioral interpretations, and high downstream utility does not by itself identify what behavioral information they encode.

Explicit, Fitted, and Learned Mapping Semantics

Representation mappings can be categorized without equating explicit with simple or learned with opaque:

Explicit Mapping: Source-to-component semantics are directly specified and declared.
Fitted Explicit Mapping: Explicit mapping with data-dependent parameters estimated from training or calibration data (e.g., scaling, PCA basis).
Projected/Factor Mapping: A fitted mapping projecting into lower-dimensional subspaces or factor representations.
Learned Encoder: Mappings substantially derived from optimization or learning processes, potentially complex and nonlinear.
Task-Specific Representation: Learned or constructed representations tailored to a particular downstream task or objective.

Learning objective, training data, augmentations, supervision, architecture, regularization, extraction point, and checkpoint can shape a learned representation's invariances, separability, retained nuisance information, and geometry. Such dependencies are part of the representation identity and interpretation rather than being uniquely defined by model family names.

Latent-coordinate identifiability and equivalence: Two fitted representations can encode essentially similar relational information while differing by permutation, sign, rotation, reflection, scaling, basis change, or other admissible transformation. Therefore, individual latent coordinates need not be historically or semantically comparable across independently fitted states. Coordinate interpretation requires evidence beyond numerical position alone.

Downstream utility, probes, reconstruction, visualization, separability, and transfer performance provide evidence about selected representational properties rather than definitions of Behavioral Representation. A representation can be scientifically useful without every coordinate being interpretable, while strong performance can also reflect participant identity, device, context, leakage, or other nuisance information that the intended representation should not preserve.

Mapping Identity	Data-Dependent State	Coordinate Interpretability	Comparability Requirement
Explicit Mapping	None	High; semantics declared directly	Identical schema and parameters
Fitted Explicit Mapping	Parameters estimated from data	Moderate; depends on parameter stability	Alignment or shared parameter definitions
Projected/Factor Mapping	Data-dependent bases or factors	Lower; bases may be arbitrary	Basis alignment or canonicalization needed
Learned Encoder	Learned parameters, weights	Variable; often partial or latent	Checkpoint/version matching and schema agreement
Task-Specific Representation	Learned with supervision	Limited to task semantics	Task and training condition matching

Information Preservation, Loss, Invariance, and Equivariance

Information preservation and loss are relation-specific properties of a mapping. A representation can preserve exact samples, relative ordering, temporal structure, geometry, event identity, selected descriptors, neighborhood relations, reconstructability, predictive information, or another declared property while discarding other information. Terms such as "lower dimensional," "compressed," "normalized," or "learned" do not specify which scientific information was preserved.

Invariance means a representation deliberately remains unchanged under a declared transformation, while equivariance means it changes according to a predictable transformation rule. Transformations such as translation, rotation, scale, time shift, speed change, amplitude gain, participant identity, or device identity can be target-relevant or nuisance depending on the scientific question. Greater invariance is therefore not universally better.

Compression, dimensional reduction, quantization, pooling, aggregation, normalization, and discretization are operations that can improve compactness or selected invariance while losing information. Loss can be intentional and scientifically beneficial when it removes nuisance variation, but an irreversible mapping should not be described as information preserving merely because downstream performance remains high for one task.

Invertibility and reconstructability are different strengths of information-retention evidence. Exact invertibility can demonstrate mathematical retention relative to the declared source object; approximate reconstruction measures what a particular decoder or inverse procedure can recover. Neither guarantees preservation of the behavioral distinction of interest. Conversely, a noninvertible representation can still preserve all information necessary for a narrower declared scientific purpose.

Comparability, Missingness, Uncertainty, and Provenance

Representation comparability and version identity require that instances are directly comparable only when the representation definition, schema, component/token/node identities, source/support semantics, parameters, fitted state, normalization, and relevant mapping conventions are compatible or an explicit alignment or mapping justifies comparison. Independently refitted bases, vocabularies, cluster IDs, graph policies, or encoder checkpoints can define different coordinate systems even when output dimensions and labels look identical.

Missingness and partial representation arise from missing source values, absent modalities or channels, invalid descriptors, occluded landmarks, unknown symbols, unavailable nodes or edges, masked sequence steps, and representation-computation failures. These are scientifically different states. Padding, imputation, learned missing tokens, masks, omission, fallback representations, or partial structures change what evidence is represented and must remain distinguishable from observed zero values or genuine absence.

Representation uncertainty and ambiguity can arise from uncertain source evidence, event or landmark boundaries, parameter choice, fitted-state estimation, stochastic mapping, latent-coordinate nonidentifiability, graph construction, symbolization, alignment, or incomplete information. It is critical to distinguish uncertainty in represented values from uncertainty about representation semantics and from uncertainty in a downstream inference that consumes the representation.

Change	Potential Identity Consequence	Comparability Requirement
Source/Support Change	Different evidence base, representation identity altered	Requires mapping or schema adjustment
Schema Change	Different interpretation of same data	Requires schema alignment or conversion
Component or Vocabulary Change	Coordinate/token identity shift or mismatch	Requires explicit alignment or transformation
Normalization Change	Value scale or distribution shift	Requires normalization agreement
Fitted Basis/Codebook Change	Coordinate system or embedding shifts	Requires basis alignment or checkpoint matching
Encoder Checkpoint Change	Learned parameters differ, representation altered	Requires checkpoint/version control
Graph Construction Change	Topology or relation definition altered	Requires explicit graph policy reconciliation
Missingness Policy Change	Different handling of missing or invalid data	Requires explicit mask or imputation semantics

Integrated Worked Example: Single Behavioral Episode Representations

Consider the same behavioral episode with the following identified evidence:

A prepared movement signal sampled over time,
A small set of descriptor instances (e.g., velocity peaks, acceleration contours),
Several detected discrete events (e.g., foot strikes, hand gestures),
A short pose trajectory of body landmarks.

This evidence can be represented in multiple forms:

Representation Form	Mapping	Schema	Support	Preserved Structure	Discarded Information	Missingness Handling	Meaningful Operation or Comparison
Fixed Feature Vector	Extract descriptors, normalize, concatenate	Fixed order, component identities	Entire episode or defined window	Component identity, magnitude	Temporal order, fine timing details	Imputation or masking	Euclidean distance for participant clustering
Time-Indexed Descriptor Sequence	Ordered descriptor extraction per frame	Sequence order, timestamps	Temporal interval	Temporal order, descriptor identity	Spatial relations, latent factors	Masked steps or missing frames	Sequence alignment or dynamic time warping
Discrete Event-Token Sequence	Event detection and token assignment	Ordered tokens, vocabulary version	Event time points	Event identity, token semantics	Continuous signal magnitude, spatial detail	Missing or uncertain tokens	Event pattern frequency or Markov modeling
Geometric Trajectory/Tensor	Pose extraction, coordinate normalization	Spatial coordinates, frame, units	Time-indexed landmarks	Spatial geometry, temporal order	Descriptor values, discrete events	Occluded landmarks masked	Trajectory matching or spatial correlation
Relational Graph	Nodes: landmarks/events; edges: relations	Node and edge identities, weights	Spatial and temporal relations	Entity relations, graph topology	Raw signal magnitudes, temporal order	Missing nodes/edges masked	Graph similarity, centrality analysis
Hypothetical Learned Latent Encoding	Encoder network applied to evidence	Latent coordinate space, learned parameters	Episode or segment	Compressed predictive info	Explicit coordinate semantics, ordering	Learned missing tokens or masks	Classification accuracy, transfer learning

Changing the representation alters the scientific interface to the evidence without changing the original observations. Each form preserves different aspects of the information and supports different operations. No single representation is universally most informative; each serves distinct scientific questions and analytical methods.