Missingness and Data Loss

Missingness and Data Loss are the scientific characterizations of expected behavioral, physiological, neurophysiological, digital, contextual, or acquisition evidence that is absent, unavailable, incomplete, irretrievable, or not validly retained over some declared support. Missingness describes the state in which expected evidence is unavailable, whereas data loss describes the process by which evidence that should have been acquired, transmitted, preserved, or retained becomes unavailable. It is crucial to establish that missing evidence is not evidence that the underlying phenomenon was absent. Furthermore, missingness is not synonymous with artifact, noise, invalidity, censoring, zero value, or participant attrition.

Meaning of Missingness and Data Loss

Missingness is defined relative to an expected observation scheme: an observation is missing only when the scientific or acquisition design establishes that evidence could or should have existed at a defined time, source, channel, event, variable, participant, spatial region, or other support but the corresponding valid record is unavailable. Absence cannot be interpreted without knowing what was expected to be observed.

Data loss is defined as the failure to preserve evidence across acquisition, transmission, buffering, persistence, synchronization, export, storage, or another retention step after the evidence was expected to become available. Not all missingness is data loss: planned non-observation, not-applicable measurements, unavailable behavioral opportunities, or deliberately unsampled periods can produce absent values without any evidence having been lost.

Expected Evidence and Observation Opportunity

Expected evidence consists of the observations that should be available under a declared acquisition schedule, event logic, participant state, source configuration, sensing opportunity, or recording condition. A missingness judgment must distinguish expected-but-unavailable evidence from evidence that was never expected under the design.

Observation opportunity refers to the conditions under which a behavioral event, physiological state, contextual condition, or digital action can be recorded. This requires that the relevant source is observable and the acquisition system is active, configured, and capable of retaining the evidence. The absence of a record should not be interpreted as event absence when the observation opportunity itself was missing or uncertain.

Phenomenon absence, source silence, sensor non-detection, unavailable observation opportunity, and missing record represent distinct conditions: a participant may produce no event while the sensor functions correctly; a phenomenon may occur below detectability; the sensor may be unable to observe it; or the event may be observed transiently but lost before persistence. These states support different scientific conclusions.

Structural and design-based absence occurs when observations are intentionally not collected because a modality, task, condition, participant role, time interval, or measurement is not applicable or not scheduled. Such absence must be distinguishable from unexpected loss so that designed sparsity is not misclassified as acquisition failure.

Forms and Patterns of Missing Evidence

Isolated missing observations and short gaps represent local interruptions in otherwise available evidence. Their scientific consequence depends on temporal resolution, event density, expected continuity, and whether a critical event or transition could have occurred inside the gap.

Contiguous gaps, burst loss, intermittent dropout, and recurrent missing intervals arise from communication failures, intermittent contact, participant nonwear, occlusion, device duty cycling, software behavior, power state, or other mechanisms. Patterns of recurrence can themselves provide evidence about the missingness mechanism.

Truncation and premature recording termination refer to loss of the beginning, end, or remaining portion of an intended acquisition interval. Causes include late start, early stop, file truncation, storage exhaustion, device shutdown, process failure, or participant interruption. Truncation can systematically remove particular phases or behaviors rather than producing uniformly distributed missingness.

Missingness can occur at various levels: source-level, channel-level, modality-level, participant-level, and session-level. One stream can fail while others continue; one participant can become unobservable while a shared recording remains active; one modality can be absent for an entire interval; or an entire acquisition occasion can be lost. Global completeness should not conceal which evidential units are missing.

Monotone missingness occurs when evidence becomes unavailable and remains unavailable over the remainder of a relevant sequence. Nonmonotone missingness allows later observations to resume. These terms describe structural patterns without assuming particular statistical missingness mechanisms.

Origins and Mechanisms of Data Loss

Sensing and interface failures create missing evidence through complete contact loss, sensor detachment, occlusion, out-of-range geometry, unavailable field of view, source disconnection, sensor shutdown, or failure to detect a source that the acquisition design expected to observe. Complete loss differs from degraded but still present evidence.

Device and power-related loss includes battery depletion, thermal shutdown, restart, firmware failure, operating-system suspension, storage exhaustion, peripheral disconnect, or acquisition-process termination. The presence of power does not guarantee that the intended signal was continuously retained.

Communication and transport loss arises from dropped packets, unavailable network links, wireless disconnection, queue overflow, retransmission failure, delayed data exceeding retention windows, or transport-layer interruption. Packet loss can cause missing samples, frames, events, duplicated recovery behavior, or irregular timing depending on the recording system.

Buffering, persistence, storage, and file-level loss result from buffer overflow, failed writes, incomplete file closure, corruption, file truncation, storage replacement, overwritten records, serialization failure, or export omission. Acquisition success at the sensor does not guarantee persistent evidence.

Human and operational sources of missingness include nonwear, device removal, failure to charge, skipped measurements, denied permissions, participant interruption, operator omission, incorrect start or stop, unavailable participant, or unperformed task or report. Such missingness should not be assumed random or independent of behavior, burden, privacy concerns, health, mobility, or context.

Missingness Mechanisms and Informativeness

Missing Completely At Random (MCAR), Missing At Random (MAR), and Missing Not At Random (MNAR) are statistical assumptions about how the probability of missingness depends on observed and unobserved data. These concepts describe the dependence structure between missingness and data rather than acquisition causes alone. Thus, acquisition causes such as device failure, nonwear, or skipped observation do not determine MCAR, MAR, or MNAR by themselves.

MCAR is the condition where missingness is independent of relevant observed and unobserved data under the stated model. Accidental technical failures are not automatically MCAR if their occurrence depends on participant behavior, context, device use, location, signal magnitude, or other variables related to the data.

MAR is the condition where, conditional on relevant observed information, missingness does not additionally depend on the unobserved value being modeled. MAR is an assumption about conditional dependence rather than a synonym for random-looking gaps.

MNAR occurs when missingness depends on the unobserved value or state after accounting for relevant observed information. For example, a wearable might be removed preferentially during a particular activity, or a participant might skip observations during scientifically relevant states. The true mechanism is often not identifiable from the observed record alone.

Informative missingness more broadly refers to missingness whose occurrence carries information about behavior, physiology, context, acquisition burden, participant choice, system state, or other scientifically relevant conditions. Missingness patterns can thus be both a quality limitation and evidence about circumstances under which observation failed, without revealing the missing values themselves.

Missingness, Invalidity, Censoring, and Corruption

Physically absent evidence differs from present-but-invalid or unusable evidence. A record may exist but fail quality requirements because it is saturated, corrupted, misattributed, temporally uninterpretable, or otherwise invalid for a purpose. Such evidence can be treated as unavailable for a particular analysis while remaining scientifically distinct from data that were never recorded or physically lost.

Censoring differs from missingness by retaining partial information about a quantity, such as knowing it lies above, below, before, after, or outside a measurable bound. Missingness provides no valid observed value for the expected quantity. Clipping, detection limits, and thresholded reporting can sometimes create censored or range-limited information rather than pure missingness.

Corrupted records are those in which a value, packet, frame, event, timestamp, or file is internally inconsistent or undecodable while a physical representation remains present. Corruption differs from loss, and whether any partial information remains recoverable should be preserved.

Explicit missing-value markers and sentinel values—such as null markers, NaN values, validity flags, impossible codes, masks, or reserved values—can represent missingness but only if their semantics are documented. A numerical zero, empty string, repeated previous value, or default value should not be assumed to mean missing without evidence.

Temporal, Spatial, and Multistream Support of Missingness

Missingness support is the exact temporal, spatial, channel, stream, participant, modality, event, or variable scope over which evidence is unavailable. A percentage missing without support information can conceal whether loss is diffuse, concentrated in critical events, confined to one participant, or clustered in one context.

Gap duration, gap location, recurrence, and boundary uncertainty affect interpretation. Two recordings with the same missing-data percentage may differ substantially if one contains many short gaps and the other loses one long interval containing a critical behavioral episode. Uncertain gap boundaries should be preserved when the exact first or last missing observation cannot be established.

Unequal missingness across channels, modalities, sources, and participants is common. Shared recording duration does not imply equal evidential coverage, and a complete primary stream can coexist with missing contextual, physiological, behavioral, or participant-specific evidence. Cross-source analyses should not silently treat partially observed combinations as complete.

Overlap and cascading loss occur when loss of a common clock, reference, participant identifier, synchronization marker, shared power source, or mapping record makes several otherwise present streams unusable together. The scientific loss can therefore exceed the number of physically missing samples.

Assessment, Severity, and Scientific Consequences

Completeness measures such as missing percentage, valid-sample fraction, coverage duration, valid-event count, available-source fraction, or participant-specific coverage can summarize missingness. However, no one denominator is universally correct. Every completeness measure should define what was expected, what counts as validly observed, and the support over which the ratio is computed.

Missingness severity is better understood in terms of information lost rather than quantity alone. A short gap can be severe if it removes a unique event, transition, synchronization anchor, or rare state, while a longer planned absence can be irrelevant to a claim whose required evidence lies elsewhere. Severity is purpose-dependent and support-dependent.

Scientific consequences of missingness include biased temporal coverage, distorted event rates, uncertain durations, lost transitions, incomplete multimodal correspondence, reduced participant comparability, broken temporal continuity, unavailable reference information, and potentially biased downstream estimates when missingness is informative. The direction and magnitude of consequences depend on what is missing and why.

Missingness characterization is distinct from imputation, interpolation, reconstruction, carry-forward, model-based completion, or reacquisition. Characterization identifies what evidence is absent and why it may be absent; completion procedures estimate or substitute values under assumptions, while reacquisition can restore later observation but cannot directly observe an interval that has already passed. Filled values should remain distinguishable from directly acquired evidence.

Missingness Provenance and Interpretation in Behavioral Signal Processing

Missingness provenance encompasses the information needed to reconstruct what evidence was expected, what became unavailable, over which support, and through which known or suspected mechanism. Relevant provenance includes expected observation schedules or event logic, validity masks, gap boundaries, participant and source state, device and power state, contact or visibility, network and storage state, planned absences, task or contextual conditions, reason codes, source-specific coverage, uncertainty about cause, and whether later values were directly observed, reacquired, interpolated, imputed, carried forward, or otherwise derived.

Understanding Missingness and Data Loss is crucial in Behavioral Signal Processing because missing evidence can alter apparent behavioral frequency, duration, timing, variability, cross-signal relationships, participant comparisons, longitudinal trajectories, contextual interpretation, and inference. Missingness is not simply an empty cell or absent sample; it is an evidential condition whose meaning depends on what should have been observed, whether observation was possible, how the evidence became unavailable, and whether the pattern of absence is itself scientifically informative.