Time-Frequency Descriptors

Time-Frequency Descriptors analyze signals by combining time and frequency data, offering insights into both temporal and spectral characteristics.

Time-Frequency Descriptors explicitly characterize how frequency-, oscillation-, or scale-dependent signal structure changes through time within declared Behavioral Signal evidence. They emerge from intermediate time-frequency representations such as Short-Time Fourier Transform (STFT) matrices, spectrograms, continuous-wavelet transforms (CWT), scalograms, filter-bank maps, reassigned maps, or related joint time-frequency (or time-scale) representations. These representations are not themselves complete descriptors; rather, a Time-Frequency Descriptor is an explicitly defined local functional, contour, change measure, ridge, region, event-aligned characterization, or structured summary derived from such a representation.

Joint localization in both time and frequency (or scale) is essential: if collapsing either axis loses information required by the intended property, then the characterization is genuinely time-frequency rather than merely spectral or temporal. Thus, Time-Frequency Descriptors depend crucially on the two-dimensional structure of the representation and its associated coordinate semantics.

Meaning and Time-Frequency Representation

A Time-Frequency Descriptor is a reproducible characterization whose scientific meaning depends jointly on an identified time-frequency or time-scale representation, its time coordinates, frequency or scale coordinates, transform and scaling conventions, declared valid region, descriptor functional, parameters, and output semantics.

Descriptor outputs may be scalar values, vectors, contours or trajectories, ridge paths, region statistics, map-derived event-like objects, distributions, or other structured values, provided these outputs have a defined and stable scientific interpretation.

Several related concepts must be distinguished:

Transform Coefficients: Complex-valued outputs of a time-frequency transform (e.g., STFT, CWT) that retain both magnitude and phase information.
Magnitude Time-Frequency Map: Nonnegative maps formed by taking the magnitude (absolute value) of transform coefficients, often visualized as spectrograms or scalograms.
Power or Energy-Like Map: Maps derived from squared magnitude coefficients, representing localized energy or power density.
PSD-Like Local Spectral Map: Maps normalized or scaled to resemble Power Spectral Density (PSD), often incorporating window normalization and spectral leakage correction.
Scalogram: A specific form of time-scale map derived from squared magnitude CWT coefficients, interpreted as localized scale-dependent energy.
Reassigned or Concentrated Map: Time-frequency maps where coordinates are adjusted based on transform phase derivatives or other estimators to sharpen localization of oscillatory components.
Time-Frequency Descriptor: An explicitly defined functional or structural summary computed from one of the above maps or coefficients, incorporating coordinate semantics and parameterization.

Complex transform coefficients preserve phase information useful for phase-based descriptors. Nonnegative magnitude or power maps support weighting, integration, and statistical measures. Normalized maps remove some absolute magnitude information to facilitate comparison across signals or conditions. Reassigned coordinates use estimator-derived localization, which may differ from the nominal transform grid.

The same descriptor functional (e.g., "peak frequency") applied to different map semantics (complex coefficients vs. magnitude spectrogram vs. reassigned map) can produce scientifically distinct quantities.

Object	Axes	Representation Meaning	Descriptor Status
Complex STFT	Time × Frequency	Complex Fourier coefficients localized by window	Transform Coefficients
Magnitude Spectrogram	Time × Frequency	Magnitude of STFT coefficients; nonnegative spectral energy density	Magnitude Time-Frequency Map
Power/PSD-Like Spectrogram	Time × Frequency	Squared magnitude, normalized for PSD-like interpretation	Power or Energy-Like Map
CWT Coefficients	Scale × Time	Complex wavelet transform coefficients indexed by scale and time	Transform Coefficients
Scalogram	Scale × Time	Squared magnitude of CWT coefficients; scale-dependent energy density	Scalogram
Filter-Bank or Constant-Q Map	Time × Frequency	Filter-bank outputs with specific bandwidth and frequency distribution	Magnitude Time-Frequency Map
Reassigned/Concentrated Map	Time × Frequency	Maps with coordinates shifted to sharpen localization of components	Reassigned or Concentrated Map
Time-Frequency Descriptor	Various	Derived local functional, contour, ridge, or region with explicit semantics	Time-Frequency Descriptor

A time-frequency map is not merely an image. Pixel or matrix indices acquire scientific meaning only through explicit time coordinates, frequency or scale coordinates, units, transform support, scaling, and declared valid regions. Image-processing operations applied to a displayed spectrogram can alter or ignore these physical coordinates. If region geometry or texture is used as a descriptor, the coordinate metric and map semantics must remain explicit to avoid misinterpretation.

Time × Frequency Time-Scale Map
Scale ↔ Frequency Mapping Time-Frequency Descriptor Families Local Spectral Contours Spectral Change Ridges Time-Frequency Regions Window / Wavelet Hop / Scale Grid Scaling Boundary Validity Provenance

Time and Frequency Localization Semantics

Time-coordinate semantics in localized analysis depend on the transform and windowing convention:

Frame start: The reference time is the first sample in the analysis window frame.
Frame center: The reference time is the center sample of the analysis window.
Frame end: The reference time is the last sample in the window.
Trailing or causal anchor: The reference time is the latest contributing sample, appropriate for causal or real-time analysis.
Custom offset: An explicitly declared offset is applied for alignment or latency correction.
Wavelet translation coordinate: The wavelet transform’s translation parameter identifies the time location of the analyzing wavelet.
Reassigned time: An estimator-derived coordinate computed from transform phase derivatives to refine localization.

A map timestamp identifies a reference coordinate for evidence with finite temporal extent; it does not imply that all contributing evidence occurred precisely at that instant. Window centering, padding, offset, and timestamp conventions must be preserved when alignment or latency matters.

Frequency- and scale-coordinate semantics include:

Hertz (Hz): Physical frequency in cycles per second.
Normalized frequency: Frequency normalized by sampling rate, often expressed as cycles per sample or radians per sample.
Angular frequency: Frequency in radians per unit time.
Linear or logarithmic frequency grids: Frequency bins spaced uniformly (linear) or multiplicatively (logarithmic).
Filter-bank centers: Center frequencies of individual bandpass filters.
Wavelet scale: The dilation parameter controlling wavelet width, related but not equal to frequency.
Pseudo-frequency: A wavelet-specific mapping from scale to an equivalent frequency.
Reassigned frequency: An estimator-derived frequency coordinate aimed at improved localization.

The sampling period and any scale-to-frequency or warped-frequency mapping required for physical interpretation must be declared explicitly. Wavelet scale or arbitrary bin index must never be interpreted directly as hertz without verified mapping.

\begin{matrix} X_{m, k} = \sum_{n = 0}^{N - 1} x_{n} w_{n - m H} e^{- i 2 π k n / N_{F}} \\ t_{m} = t_{0} + m H Δt \\ f_{k} = \frac{k}{N_{F} Δt} \end{matrix}

Here, N is the finite input-sample count used in the sum, n is the sample index, x_n is input sample n, w is the declared analysis window with the shown shifted indexing convention, m is the frame index, H is the hop size in samples, N_F is the transform length, i is the imaginary unit, X_{m,k} is the complex STFT coefficient at frame m and frequency bin k, t_0 is the declared time origin for the frame indexing, Δt is the sampling period, t_m is the frame-reference time, k is the frequency-bin index, and f_k is the corresponding conventional DFT-bin frequency.

Window centering and padding conventions can shift the exact relation between t_m and the observed samples.

The time–frequency localization tradeoff arises because shorter effective analysis windows improve temporal localization but broaden spectral localization or reduce discrimination between nearby frequencies. Conversely, longer windows improve frequency discrimination but average over longer temporal structures.

A smaller hop size increases the sampling density of the time-frequency map without shortening the underlying analysis window. Zero-padding can densify the frequency grid but does not create the resolution of longer observed evidence.

Overlap and dependence between localized frames must be acknowledged: Adjacent STFT windows, neighboring wavelet translations, and highly redundant filter-bank outputs share substantial source evidence. Dense time-frequency contours therefore contain dependent descriptor instances. More map columns or coefficients do not correspond to an equal number of independent behavioral observations.

Configuration	Temporal Localization	Frequency Characterization	Interpretive Consequence
Short Window	High (narrow window)	Low (broad frequency response)	Better temporal precision but poorer frequency resolution
Long Window	Low (wide window)	High (narrow frequency response)	Better frequency discrimination but temporal averaging
Small Hop	Denser sampling of frames	Same as window	More data points, dependent observations, no improved resolution
Large Hop	Sparser sampling	Same as window	Less data, potential aliasing in time
Zero-Padded Transform	Same temporal localization	Denser frequency grid	No actual increase in frequency resolution; interpolation only
High Frame Overlap	High temporal sampling	Same as window	Increased correlation between frames
Causal/Trailing Window	Reference time at window end	Same as window	Latency preservation, suitable for online applications

Time-Frequency Representation Families and Valid Regions

STFT- and spectrogram-based representations are localized Fourier analyses parameterized by sampling rate, analysis window, hop size, transform length, centering, padding, one- or two-sided convention, detrending, and scaling (magnitude, power, PSD-like). Each frame characterizes a finite neighborhood of the signal, and boundary frames depend partly on padding or incomplete observed support. These parameters must be declared and preserved for valid descriptor interpretation without delving into STFT reconstruction or FFT algorithm details.

The Continuous Wavelet Transform (CWT) is a localized analysis indexed by translation (b) and scale (a), with the wavelet family and scale controlling the temporal extent and frequency sensitivity of the analyzing function. Distinct outputs include complex wavelet coefficients, coefficient magnitude, squared magnitude (scalogram), and normalized variants.

Larger scale generally corresponds to a stretched wavelet and lower characteristic frequency for commonly used wavelets, but the exact scale-to-frequency mapping is wavelet-specific and must be verified explicitly.

\begin{matrix} W_{x} (a, b) = \frac{1}{\sqrt{a}} \int x (t) ψ^{*} (\frac{t - b}{a}) d t \\ f_{a} = \frac{f_{ψ}}{a Δt} \end{matrix}

Here, x(t) is the analyzed signal in continuous time notation, ψ is the declared analyzing wavelet, ψ* its complex conjugate, a > 0 is the wavelet scale, b is the translation/time coordinate, W_x(a,b) is the wavelet coefficient, Δt is the sampling period when relating discrete implementation to physical frequency, f_ψ is a declared wavelet-specific normalized center frequency quantity under the chosen convention, and f_a is the corresponding conceptual pseudo-frequency.

Exact scale-to-frequency mapping depends on wavelet definition and implementation and must be verified, not inferred from scale alone.

Scale selection and wavelet-specific sampling affect which oscillatory structures can be represented. The chosen scale range, scale spacing, wavelet bandwidth and center-frequency parameters, sampling period, and discrete implementation determine representability. Very low scales may be undersampled or alias for some wavelets; very high scales have long effective support and strong boundary influence. A denser scale grid does not create new information beyond the analyzed evidence and wavelet resolution.

Linear filter-bank, logarithmic-frequency, constant-Q, variable-Q, and other multiresolution representations conceptually differ by their frequency-dependent bandwidth and spacing. Frequency-dependent bandwidth can be useful when relevant structure spans several frequency scales. However, logarithmic or musical frequency spacing is not universally meaningful for movement, physiological, gaze, or other nonaudio signals. Representation selection should follow target dynamics and interpretability rather than visual appearance.

Reassignment and synchrosqueezing-like concentration estimate improved local time and/or frequency coordinates from transform information to sharpen oscillatory components under method-specific assumptions. Sharper maps do not create new observed events or exact physical trajectories. Low-energy or multi-component regions can produce undefined or unreliable reassigned coordinates.

Representation Family	Localization Structure	Defining Parameters	Primary Descriptor Risk
STFT / Spectrogram	Time-localized Fourier coefficients	Sampling, window, hop, transform length, centering, scaling	Boundary effects, window choice bias, padding artifacts
CWT / Scalogram	Scale-localized wavelet coefficients	Wavelet family, scale grid, translation, sampling period	Scale-frequency ambiguity, edge effects, wavelet dependency
Filter Bank	Frequency-band outputs from filters	Filter center frequencies, bandwidths, spacing, sampling	Overlapping bands, nonuniform resolution, filter shape effects
Constant-Q / Variable-Q	Logarithmic or adaptive resolution	Q factor, frequency range, filter design	Misinterpretation of pitch or scale in nonmusical signals
Reassigned / Concentrated	Estimator-derived localization	Phase derivatives, reassignment parameters, thresholding	Unreliable coordinates in low-energy regions, multi-component ambiguity

Local Spectral Contours and Band Activity

Local spectral functionals are time-indexed applications of frequency-distribution descriptors to each valid localized spectral slice or corresponding frequency-mapped time-scale slice.

Representative contours include local band content, peak frequency, spectral centroid, spread, median frequency, roll-off frequency, spectral shape, entropy, and flatness. Each contour inherits the transform’s finite temporal support, frequency grid, scaling, normalization, overlap, and boundary conditions.

B_{m, R} = \sum_{k \in K_{R}} Q_{k, m} {Δf}_{k}

Here, m is the localized time/frame index, R is a declared frequency region or band, K_R is the set of frequency-bin indices admitted to that region under a declared edge convention, k is a frequency-bin index, Q_{k,m} is a nonnegative density-like localized spectral quantity at bin k and frame m, Δf_k is the represented bin width, and B_{m,R} is the localized band content for frame m.

A scale-domain analog requires appropriate scale weighting or a justified scale-to-frequency mapping and is not obtained by blindly summing wavelet coefficients.

Absolute and relative local band-content contours, band-ratio contours, band occupancy, and thresholded band-activity episodes require declaration of denominator, normalization range, threshold, minimum duration, and boundary treatment.

A rise in relative band content can occur because the target band increases, other bands decrease, or normalization changes; it should not be interpreted automatically as an absolute signal-amplitude increase.

Time-varying local spectral location and shape descriptors such as centroid, spread, median frequency, roll-off, skewness, slope, entropy, and flatness are contours rather than support-level constants. The contour sampling interval differs from the temporal extent of each local estimate. Rapid fluctuations in contours can reflect short windows, estimator variance, overlapping support, or low spectral weight rather than true rapid behavioral change.

Descriptor	Local Input	Time-Varying Property	Key Dependency
Local Band Content	Nonnegative spectral map	Energy or power in frequency band	Frequency bin definition, normalization
Local Peak Frequency	Magnitude or power map	Frequency of local maximum	Noise sensitivity, spectral resolution
Centroid Contour	Spectrum or scalogram	Spectral center of gravity	Weighting, scaling, frequency grid
Spread/Bandwidth Contour	Spectrum or scalogram	Spectral spread or variance	Frequency scaling, normalization
Roll-Off / Median Frequency	Spectrum or scalogram	Frequency below which % energy lies	Threshold definition, frequency grid
Local Spectral Entropy	Normalized spectrum	Spectral flatness or disorder	Normalization, frequency resolution
Local Flatness	Spectrum or scalogram	Ratio of geometric to arithmetic mean	Scaling, noise influence

Aggregation of time-frequency contours must be cautious. Mean, quantile, occupancy, slope, distribution, event-aligned average, or other summaries across local descriptor instances produce higher-level characterizations but may discard when spectral properties occurred. Aggregation of local descriptor values differs from computing a support-level spectral descriptor directly because nonlinear operations, normalization, variable validity, and overlapping evidence can produce non-equivalent results.

Spectral Change, Flux, and Transient Structure

One representative Euclidean spectral-change descriptor is:

F_{m} = \sqrt{\sum_{k \in K} {(q_{k, m} - q_{k, m - L})}^{2}}

Here, m is the localized-frame index, K is the declared aligned frequency-bin set, k is the frequency-bin index, q_{k,m} is the declared spectral quantity or normalized spectral weight at frequency bin k and frame m, L is a positive integer frame lag, and F_m is the Euclidean spectral change over that lag.

Other spectral change descriptors include L1 distance, cosine similarity, positive-only flux, divergence-like measures, frequency-transport metrics, and band-limited changes, each with normalization options that make the descriptor sensitive to spectral redistribution rather than absolute magnitude.

Spectral flux and local spectral distance families vary by representation, frame lag, frequency alignment, normalization, distance metric, rectification, and frequency weighting. Positive-only flux emphasizes spectral increases; Euclidean or L1 measures quantify absolute redistribution; cosine-type measures reflect shape change; frequency-aware transport-like distances treat small frequency shifts differently from unrelated redistribution.

Spectral-change contours differ from event detection: thresholding or peak-picking a flux or change contour creates a separate signal-event set whose count, timing, duration, or latency can be characterized. A large spectral-change value indicates transient spectral reorganization but does not establish a behavioral onset, state transition, or reference event.

Transient time-frequency regions are localized patches or connected evidence concentrations defined by thresholding, segmentation, ridge neighborhoods, event anchors, or other declared rules. Representative descriptors characterize region onset/offset, duration, frequency extent, integrated content, centroid, concentration, orientation or drift in the time-frequency plane, and occupancy. Region descriptors inherit thresholding, connectivity, coordinate metric, scaling, and boundary assumptions.

Object	Established By	What It Characterizes	Critical Non-Equivalence
Spectral Flux/Change Contour	Frame-to-frame spectral difference	Magnitude of spectral redistribution	Not a direct event; continuous measure
Thresholded Change Candidate	Flux thresholding	Candidate transient event regions	Threshold-dependent, may be false positives
Transient Time-Frequency Region	Region segmentation	Localized transient spectral activity	Depends on segmentation rule and map scaling
Map-Derived Event	Event detection algorithms	Detected spectral or oscillatory events	Requires separate event validation
Behavioral Event	Independent annotation	Actual behavioral onset or state transition	May not align temporally with map events

Time-Frequency Ridges and Component Trajectories

A time-frequency ridge is a time-indexed locus of frequency or scale positions selected under a declared local-maximum, energy, phase-derived, reassigned, wavelet, or other component criterion.

Ridge identity can require amplitude thresholds, continuity, slope or curvature constraints, allowed gaps, frequency range, component assignment, and handling of crossings. Selecting the largest bin independently at every frame is insufficient to define one coherent ridge when component identity matters.

Ridge-frequency and scale-trajectory descriptors include instantaneous/local ridge coordinate, median or mean ridge frequency, range, drift, slope, curvature, variability, residence in frequency bands, and duration of a coherent ridge.

Frequency drift through time differs from spatial trajectory geometry. A ridge estimate is not the same as a directly observed oscillator frequency but an extracted feature dependent on map resolution and noise.

Ridge strength and support descriptors include integrated ridge content, peak ridge content, ridge-to-background ratio, valid ridge duration, gap fraction, confidence or localization uncertainty, and component occupancy. These depend on ridge-neighborhood width, map scaling, background definition, and continuity rules. Strong ridges can arise from artifact or interference as well as the intended signal process.

Multi-ridge and component ambiguity occur when several simultaneous oscillatory components generate multiple ridges, crossings, merges, splits, or unresolved regions. Component identifiers, assignment uncertainty, track births/deaths, crossings, and unresolved correspondence must be preserved scientifically rather than forcing every frame into one dominant-frequency trajectory.

Ridge Form	Selection Basis	Useful Descriptor	Primary Failure Mode
Local-Maximum Ridge	Local maxima in magnitude map	Ridge frequency trajectory	Noise-induced false maxima
Energy-Weighted Ridge	Weighted average frequency	Smoothed ridge coordinate	Blurring multiple components
Reassigned Ridge	Reassigned coordinates	Sharpened ridge localization	Unreliable in low energy
Wavelet Ridge	Wavelet coefficient maxima	Scale-frequency trajectory	Scale-frequency mapping ambiguity
Multiple Component Tracks	Multi-ridge tracking algorithms	Component assignment, crossings	Track identity ambiguity

Time-Frequency Concentration, Event Alignment, and Scale-Dependent Activity

Time-frequency concentration and dispersion characterize how nonnegative map weight is distributed jointly over valid time-frequency coordinates.

Descriptors include occupied area, concentration ratio, two-dimensional centroid/spread under declared coordinate normalization, entropy of normalized map weight, sparsity, and energy concentration within selected regions.

These descriptors depend on time/frequency axis units, map resolution, bin area, scaling, thresholding, and valid-region masking. Raw pixel counts are not physically comparable when grid density differs.

Event-aligned time-frequency descriptors characterize spectral properties indexed relative to independently identified events or anchors. Outputs can include pre/post band changes, peak response time, peak response frequency, region content within event-relative windows, ridge shifts, or averaged event-relative maps and contours.

Anchor identity, timing uncertainty, event selection, overlap, baseline rule, and whether the same event information was used to tune the descriptor must be preserved because conditioning on selected events creates dependence and can introduce circularity.

Scale-dependent and multiresolution activity descriptors are especially relevant for wavelet or variable-bandwidth representations. They characterize when activity occurs at selected scales or pseudo-frequencies, how activity migrates across scales, persistence across neighboring scales, or scale-specific concentration.

Wavelet-scale occupancy must not be equated with physical-frequency occupancy unless the scale-to-frequency mapping and weighting are scientifically defined.

Time-Frequency Adequacy, Sensitivity, and Provenance

Uncertainty, invalid states, online status, and sensitivity are critical considerations for Time-Frequency Descriptors. Sources include finite support, window or wavelet localization, frame overlap, boundary padding, spectral leakage, transform scaling, low map weight, frequency/scale discretization, insufficiently sampled wavelet scales, reassignment failure, ridge ambiguity, region thresholds, event-anchor uncertainty, missing samples, irregular timing, preprocessing, and nonstationary regime mixing.

It is essential to distinguish:

An undefined ridge from zero frequency.
An invalid boundary region from absent activity.
A low-energy reassignment NaN from a measured coordinate.
A provisional causal/trailing descriptor from a completed centered or future-inclusive descriptor.

Comparisons across plausible windows, hops, transforms, wavelets, scales, frequency grids, normalizations, reassignment settings, ridge/region criteria, flux definitions, event-alignment choices, and valid-region masks expose material method sensitivity.

Worked Example and Provenance Audit

Consider a behavioral movement signal exhibiting a dominant oscillatory component increasing in frequency over time, containing one short transient event.

Support-level spectrum: A global spectrum averaged over the entire signal may show a broad peak corresponding to dominant oscillations but loses temporal evolution and transient details.
Localized STFT / Spectrogram: Applying STFT with a moderate-length window reveals temporal evolution of dominant frequency, showing upward frequency drift and transient spectral changes.
Local band-content contour: Computing band content within a frequency region corresponding to expected oscillations shows increasing energy and transient peaks aligned with the transient.
Spectral-change contour: Computing Euclidean spectral flux across frames highlights the transient spectral reorganization.
Ridge identification: Extracting a ridge trajectory in the spectrogram tracks the dominant oscillatory frequency drift.
Transient time-frequency region: Thresholding the spectral-change contour and connected component analysis isolates the transient event region, characterized by onset, duration, frequency extent, and integrated content.
CWT / Scalogram comparison: A continuous wavelet transform with declared scale-to-frequency mapping corroborates the frequency drift and transient but emphasizes scale-dependent activity and boundary effects.
Window and hop effects: Shorter windows improve temporal localization but blur frequency; longer windows improve frequency resolution but average transient dynamics. Smaller hop sizes increase time-frequency map sampling density without changing analysis window.
Boundary and low-energy considerations: Regions near signal edges or with low amplitude show increased uncertainty and reduced descriptor reliability.

All descriptor definitions, versions, source signal and preprocessing states, support, sampling and timing, transform family and version, window or wavelet parameters, hop or scale grid, transform length, frequency/scale coordinates, scaling and normalization, centering and padding, valid region, descriptor functionals and parameters, ridge/region/event definitions, implementation version, uncertainty, and sensitivity findings must be preserved to ensure computational reproducibility and to avoid overinterpretation of behavioral validity.

This example illustrates that computational reproducibility of a map and descriptor does not by itself establish behavioral validity; explicit provenance and sensitivity analysis are essential.