Evaluation Design and Data Partitioning

Evaluation Design and Data Partitioning is the scientific responsibility of constructing an evaluation protocol whose evidence is appropriate to the intended behavioral-inference claim and sufficiently independent of model fitting, tuning, preprocessing-state estimation, calibration, threshold selection, and other development decisions. It is essential to recognize that terms such as training set, development set, validation set, calibration set, test set, holdout, cross-validation fold, group split, temporal split, stratification, random split, external evaluation, and independence are not synonyms. Each describes distinct concepts or roles in the evaluation process. A partition is scientifically valid only relative to the dependence structure and the claim it is intended to support; random separation of exported samples alone does not automatically produce valid evaluation evidence.

Meaning and Boundaries of Evaluation Design and Data Partitioning

An Evaluation Design is the pre-specified relation among the inferential claim, evaluation unit, eligible population or corpus, model-development roles, partitioning rules, independence requirements, timing or availability constraints, references, decision rules, and final evaluation evidence. It articulates how all these elements interplay to ensure that evaluation results legitimately support the intended scientific conclusions.

Data Partitioning is one component of evaluation design. It involves dividing data into subsets with assigned roles (e.g., training, development, test), but by itself does not establish validity. Validity depends also on preventing leakage through preprocessing, reference construction, thresholding, model selection, or reporting decisions that might allow information to leak across partitions.

Model-fitting, development/tuning, calibration, model selection, and final evaluation are distinct phases with unique scientific roles:

• Model Fitting estimates model parameters from training data.
• Development/Tuning uses development evidence to select features, representations, architectures, hyperparameters, checkpoints, stopping rules, thresholds, or other model specifications.
• Calibration estimates mappings from model outputs to probabilities or decision thresholds to adjust predictive confidence.
• Model Selection chooses the best model among candidates based on performance evidence.
• Final Evaluation judges the resulting frozen or explicitly specified procedure on evidence not used for any prior decisions.

Evaluation designs such as internal holdout, cross-validation, external evaluation, prospective/chronological evaluation, and use-condition evaluation conceptually answer different scientific questions depending on what changes between development and evaluation evidence. None is universally superior without reference to the intended claim, sampling frame, and available data.

Evaluation design should be constructed backward from the intended claim. For example:

• If the claim concerns unseen participants, participant-level independence must be preserved wherever participant information could affect fitting or model choice.
• If the claim concerns future observations, future evidence and future-derived statistics must remain unavailable during development.
• If the claim concerns new devices, sites, dyads, or corpora, the corresponding grouping must be respected.

The partition axis must match the claimed novelty to ensure the evaluation evidence appropriately supports the inferential claim.

Evaluation Units, Dependence, and Partitioning Level

The evaluation unit is the entity over which predictions, references, losses, or evaluation outcomes are meaningfully computed or aggregated. Examples include sample, window, event, episode, trial, session, participant, dyad, group, device, site, or corpus. The evaluation unit is distinct from the partition unit: predictions might be scored at a fine level (e.g., windows), but partitioning must occur at a level that preserves independence (e.g., participant or trial) relevant to the claim.

Hierarchical and clustered dependence exist when evaluation units nest within one another or share resources. For instance, windows can be nested within episodes, episodes within sessions, sessions within participants, and participants within dyads or sites. Shared sensors, references, preprocessing states, contexts, or annotation sources can induce dependence even among nominally distinct units. Therefore, unique row identifiers do not guarantee independence.

Source-evidence identity demands that partitioning be reasoned from the underlying raw observations and dependency graph, not only from exported rows or features. Multiple windows, augmentations, representations, modalities, transformed copies, or labels derived from the same raw segment remain strongly dependent even when their row IDs differ.

Repeated-measures dependence arises from multiple observations of the same participant sharing stable traits such as physiology, speaking style, behavior, device placement, environment, or annotation patterns. A random split at the sample level that places the same participant's data into both development and evaluation partitions answers a within-participant interpolation question, not an independent unseen-participant generalization claim.

Dyadic and group dependence occurs when samples from one interaction pair or group include partner-specific coordination, shared context, task history, group roles, or relational signatures. Participant-disjoint splitting alone may still leak interaction-specific information if a participant or interaction group appears across partitions inconsistently with the intended dyad/group generalization claim.

Device, site, corpus, and acquisition-setup dependence arises when repeated use of the same hardware, setup, room, protocol, annotators, preprocessing pipeline, or corpus creates shared signatures that remain after participant separation. Partitioning must block the dependence relevant to the claim rather than assuming participant separation solves all evaluation problems.

Data Leakage and Development-State Contamination

Evaluation leakage is any pathway by which evidence that should be unavailable under the intended evaluation claim influences the fitted system, model specification, reference, threshold, calibration, data transformation, selection procedure, or reported conclusion. Leakage can occur without explicitly training on test labels and may involve unlabeled evaluation inputs, shared statistics, duplicated source evidence, or adaptive human decisions.

Overlapping-window and temporal-neighbor leakage occurs when windows derived from overlapping or adjacent raw intervals appear in both development and evaluation partitions, placing nearly duplicate behavioral evidence on both sides. This is especially problematic when segmentation is performed before grouping. To prevent this, assign source trials, sessions, participants, or required groups before generating cross-partition windows.

Preprocessing leakage arises when normalization statistics, imputation parameters, artifact thresholds, reference baselines, filters with learned state, dimensionality reductions, feature selectors, codebooks, learned embeddings, augmentation policies, or other fitted transformations are estimated using evidence from evaluation data. Applying a learned transform to evaluation data is valid only if the transform was fit exclusively on permitted training or development evidence.

Representation and feature-selection leakage happens when descriptors, latent dimensions, representation checkpoints, channels, modalities, variables, or features are chosen after inspecting final evaluation outcomes, making evaluation outcomes part of model development. Feature selection performed before cross-validation on the entire dataset leaks target or distributional information into every fold.

Target, reference, and annotation leakage occurs when evaluation references are constructed, weighted, adjudicated, hardened, or selected using predictions or errors from the system they are intended to judge, unless the resulting dependence is explicitly part of the evaluation design. Examples include model-assisted annotation, test-informed item exclusion, or label refinement contaminating apparently independent test evidence.

Threshold, calibration, and stopping-rule leakage happens when operating thresholds, calibration mappings, early-stopping checkpoints, postprocessing constants, confidence cutoffs, and abstention rules are chosen using final evaluation outcomes, making those outcomes development evidence. Separate evidence or nested procedures should be preserved when these choices must be estimated.

Duplicate, augmentation, and near-duplicate leakage arises from exact duplicates, transformed copies, augmented variants, repeated clips, temporally shifted versions, mirrored signals, or multiple modality representations of the same source episode appearing across partitions. All derivatives of one source entity must be grouped consistently when the intended claim requires source-level independence.

Grouped, Temporal, and Hierarchical Partitioning

Group-disjoint partitioning assigns all observations from a declared dependency group to one partition or fold. Groups can be participants, sessions, trials, dyads, families, devices, sites, or other scientifically meaningful clusters. Group separation is appropriate when the intended claim requires novelty at that group level; it should not be applied mechanically when within-group personalization is the intended use condition.

Participant-independent evaluation withholds entire participants from relevant development evidence and supports an unseen-person claim. Participant-dependent evaluation assesses personalized or within-person inference when that matches the use condition. Neither is universally correct or incorrect; they support different scientific claims.

Session- and trial-disjoint partitioning hold out entire sessions or trials, blocking within-unit leakage but not automatically establishing unseen-participant generalization. Participant-disjoint splitting may subsume session separation if every participant belongs to only one partition.

Chronological and prospective partitioning respect temporal ordering: development evidence precedes evaluation evidence, and future observations or future-derived normalization statistics remain unavailable during development. Random shuffling can violate causal availability even if rows are otherwise independent.

Temporal gaps, embargoes, or buffer regions reduce leakage from overlapping windows, smoothing, delayed labels, autocorrelation, or feature construction near split boundaries. The necessary gap width depends on the source support and dependency mechanism; no universal embargo size exists.

Multi-axis partitioning involves simultaneous separation by multiple axes such as participant and site, participant and session, dyad and time, or device and participant. It is critical to state which axes are held disjoint and which are allowed to repeat. Blocking one dependence axis may leave others scientifically material independent.

Holdout, Cross-Validation, and Nested Evaluation Designs

A fixed holdout design reserves one or more partitions for evaluation while development occurs on the remainder. Holdout evaluation is simple and can preserve clear final independence, but performance estimates can depend strongly on which groups or cases were held out when data are limited or heterogeneous.

Cross-validation performs repeated partitioning so that each eligible unit or group contributes to evaluation under a declared fold structure. Cross-validation does not guarantee immunity to leakage: preprocessing, feature selection, hyperparameter tuning, group violations, temporal leakage, or reference reuse can contaminate every fold if not confined to the appropriate training portion.

Grouped cross-validation maintains integrity of dependency groups, preventing groups from appearing in both training and evaluation within the same fold. Group balance and class balance can conflict, especially with few groups or rare behaviors. Preserving independence must take precedence over cosmetic sample-level balance when group novelty is required.

Temporal cross-validation respects chronological availability by using earlier observations for development and later observations for evaluation. Successive folds can use increasing training history, fixed or expanding windows, and optional gaps to block leakage. Future observations must not estimate states applied to earlier evaluation intervals.

Nested cross-validation handles model specification or hyperparameter selection when performance is estimated on the same finite dataset. The inner procedure tunes or selects using only outer-training evidence; the outer evaluation estimates performance of the full selection procedure. Non-nested optimization and evaluation on the same folds can produce optimistic selection bias.

Repeated splits or repeated cross-validation characterize dependence of results on partition assignment when random grouping is scientifically permissible. Repetition does not create new independent data, and scores from repeated folds or splits are statistically dependent because they reuse observations. Reporting number of folds or repeats as independent samples is incorrect.

Stratification, Coverage, and Partition Feasibility

Stratification attempts to preserve selected distributions (e.g., target classes, outcome prevalence) across partitions while respecting stronger independence constraints. Stratification can improve fold usability and comparability but does not guarantee representative sampling, independent observations, balanced groups, or unbiased population prevalence.

Rare behaviors, small groups, and sparse strata create challenges. Some participants, sessions, or sites may contain only one class or very few target events, making perfectly stratified group-disjoint folds impossible. Scientific grouping constraints must be preserved, and resulting imbalance, unsupported folds, or reduced fold counts should be reported rather than splitting dependent groups merely to force numerical balance.

Coverage and eligibility must be specified by stating which participants, sessions, behaviors, contexts, devices, modalities, and references are eligible for each partition and why. Excluding difficult or incomplete cases after seeing evaluation outcomes changes the evaluated population and must be pre-specified or transparently treated as sensitivity analyses.

Representativeness versus independence are distinct properties. A perfectly independent holdout can be unrepresentative of the intended use population, and a representative sample can still leak information into development. Evaluation design should justify sampling coverage and independence separately rather than treating either as a substitute for the other.

Protocol Freezing, Reuse, and Evaluation Provenance

Repeated inspection and adaptive overfitting to evaluation evidence occur when researchers repeatedly examine scores, errors, subgroups, examples, or leaderboard positions from a nominal test set to choose models, features, thresholds, reporting strategies, or narratives. Such practice causes the test set to lose confirmatory status without any direct gradient update on its labels.

Protocol freezing and reproducibility require preserving, before final evaluation, the split-generating rule, grouping variables, random seed (if randomness is used), eligible-data version, preprocessing fit boundaries, model-selection procedure, calibration procedure, evaluation reference version, and stopping or threshold rules. Re-running a documented randomized split can be reproducible but still answer the wrong scientific question; reproducibility does not replace design validity.

Integrated Worked Example

Consider multimodal behavioral data collected from repeated sessions of multiple participants across two devices and two sites.

• A naive random window split places overlapping windows from the same trial on both sides, causing raw-support overlap leakage.
• A corrected participant-disjoint split is created to support an unseen-person claim, blocking participant identity leakage.
• A within-participant chronological design serves personalization use, respecting temporal availability.
• Normalization is fitted only on allowed training evidence to prevent preprocessing leakage.
• Feature selection and calibration are confined to development data to avoid representation and calibration leakage.
• Nested model selection is performed inside outer participant-held-out folds to prevent optimistic bias.
• Class stratification is applied constrained by participant groups, but one rare class precludes perfectly balanced folds.
• The test set loses confirmatory status after repeated threshold tuning, illustrating adaptive test inspection.
• An external-site evaluation is performed; stronger site novelty does not automatically imply future-time or new-device generalization without explicitly blocking those dependencies.

Evaluation Design and Data Partitioning provenance requires preserving all information needed to reproduce and scientifically interpret the evaluation protocol when material. This includes:

• Intended inferential claim
• Target and evaluation unit
• Eligible population or corpus
• Source-data version
• Hierarchy and dependency groups
• Partition unit and grouping axes
• Train/development/calibration/test roles
• Holdout or cross-validation scheme
• Fold definitions
• Temporal ordering and gaps
• Stratification variables
• Random seed and split algorithm (if used)
• Overlap and duplicate controls
• Preprocessing and representation fit boundaries
• Feature-selection and model-selection procedures
• Nested structure (if used)
• Threshold and calibration fitting evidence
• Reference-construction independence
• Exclusion rules
• External-evaluation provenance
• Test-reuse history
• Protocol-freeze point
• Missingness and coverage
• Sensitivity analyses
• Implementation and version
• Limitations

A defensible evaluation design explicitly states what novelty the evaluation evidence represents, which dependencies were blocked, which development decisions were permitted to see which data, and why the resulting performance estimate is appropriate for the intended behavioral-inference claim.