19.13 Decision Noise

Decision noise is unwanted variability in judgments and decisions that should, by any rational account, produce the same outcome. Where bias causes decisions to deviate systematically in one direction — always too high, always favoring one group, always underestimating risk — noise causes decisions to scatter unpredictably around whatever the correct judgment would be. Two decision makers facing identical cases reach different conclusions; the same decision maker facing the same case at different times reaches different conclusions; different members of the same institution, trained in the same procedures, applying the same stated criteria, produce divergent outcomes. This variability is decision noise: not a pattern that points in a single direction and can be corrected by adjustment, but an irreducible dispersion that undermines consistency, fairness, and predictability.

Noise and Bias Distinguished

The distinction between noise and bias is foundational to understanding why noise is both underrecognized and difficult to address. Bias is the systematic component of decision error: it shifts outcomes consistently in one direction. Noise is the random component: it scatters outcomes around whatever the systematic component would produce. In a diagnostic analogy, bias is like a scale that always reads three kilograms heavy — it produces wrong readings, but they are consistently wrong in the same direction. Noise is like a scale whose readings fluctuate unpredictably — the average may be right, but individual readings are all over the place.

Organizations are much more aware of bias than of noise. Bias produces visible patterns: consistently lenient sentences for certain defendant types, consistently inflated valuations of certain asset classes, consistently underestimated project costs. These patterns can be identified and addressed through policy changes, training, or process redesign. Noise produces no visible pattern — just case-by-case scatter that looks like each individual decision maker exercising legitimate judgment. The noise is invisible without specifically designed audits that present identical cases to multiple judges or track the same judge's decisions over time.

Types of Decision Noise

Noise in organizational decision systems takes several forms:

Level noise is variability between individuals in their overall tendencies — some judges give higher sentences than others on average, some underwriters price risk higher than others, some doctors recommend more aggressive treatment than others — independent of the specific characteristics of the cases they handle. Level noise reflects persistent differences in the calibration of different decision makers to the same implicit standard.

Pattern noise is variability in how different individuals weight different factors in their decisions. Two underwriters might have similar average prices but respond differently to the same risk factors — one weighting the applicant's age heavily and another weighting the coverage amount heavily. Pattern noise produces inconsistencies not in overall level but in the structure of decisions.

Occasion noise is variability in the same individual's decisions across different times and contexts. A doctor who has just seen a patient die from under-treatment may be more aggressive in a subsequent similar case than they would otherwise be; a judge who has heard a difficult case before lunch may be more lenient after eating; an underwriter facing many consecutive borderline applications may drift in their assessments. Occasion noise reflects the sensitivity of human judgment to contextual factors that should, by any principled account, be irrelevant to the decision.

Noise in Organizational Decision Systems

Organizations that delegate decision-making authority to multiple people — as every large organization does — are noise-generating machines. Each decision maker brings their own implicit calibration, weighting, and sensitivity to contextual factors. The result is that nominally identical cases receive substantially different treatment depending on which decision maker handles them. This variability undermines:

Fairness: People with identical cases receive different outcomes based on the accident of which decision maker they encounter. This distributional unfairness is often invisible because those who receive different treatment do not know what outcome a different decision maker would have reached.

Predictability: When decisions are noisy, the outcome of a case cannot be reliably predicted from its characteristics. This unpredictability complicates planning, investment, and behavior adaptation by those who must anticipate institutional decisions.

Efficiency: Noisy decision systems produce more errors in aggregate than low-noise systems calibrated to the same average level. The scatter around the target outcome means that some cases will be far from the optimal decision in each direction, whereas low-noise systems concentrate outcomes near the target.

Noise Audits and Measurement

Noise can be measured through structured noise audits: presenting the same cases to multiple decision makers independently, and measuring the dispersion of their outcomes. The dispersion — commonly measured as the coefficient of variation or the standard deviation of outcomes across decision makers — quantifies the noise level. Organizations that have conducted such audits in fields as diverse as forensic analysis, medical diagnosis, insurance underwriting, and personnel evaluation have consistently found noise levels that are both large and surprising to the professionals involved.

These audits are psychologically uncomfortable for professionals who believe their judgment is reliable and consistent, because they reveal that similarly trained, well-intentioned professionals reach substantially different conclusions when presented with the same cases. Accepting the existence of high noise levels is a precondition for addressing them.

Noise Reduction Strategies

Several strategies can reduce decision noise without eliminating the human judgment that is valuable in complex decisions:

Structured judgment protocols break decision tasks into explicit components, requiring decision makers to assess each component before combining them into an overall judgment. This structure reduces the influence of first impressions and reduces occasion noise by ensuring that relevant considerations are systematically considered.

Calibrated scales and anchors provide shared reference points that help multiple decision makers interpret assessment scales consistently. When decision makers all understand what a rating of 7 means on a given dimension, level noise is reduced.

Aggregation and averaging: Combining the independent judgments of multiple decision makers — rather than treating the first available judgment as final — averages out individual noise and produces more stable, consistent outcomes.

Delayed holistic evaluation: Making holistic evaluations only after gathering and evaluating all the relevant information, rather than forming a global impression first and then interpreting information in its light, reduces the influence of irrelevant contextual factors on the holistic judgment and reduces occasion noise.

The goal of noise reduction is not to eliminate judgment but to channel it: to ensure that the variable parts of human judgment track the features of cases that are genuinely relevant to good decisions, rather than reflecting the accidental features of the moment, the decision maker's mood, or irrelevant aspects of the decision context.