23.14 Predictive Control Concern

Predictive control concern describes the set of ethical, political, and social risks that arise when surveillance systems move beyond recording past behavior to predicting future behavior and using those predictions as the basis for preemptive interventions — controlling individuals not for what they have done but for what algorithmic analysis suggests they are likely to do. In classical surveillance-based control, behavior is observed, evaluated against a standard, and corrective feedback is applied when a violation is detected. In predictive control, the feedback loop is extended or shortcut: the system predicts that a violation will occur and intervenes before it happens, based on behavioral patterns, profile characteristics, or statistical models that associate certain attributes with elevated risk. The concern is not that prediction is unreliable — it often is — but that the paradigm of predictive control itself changes the relationship between the state, institutions, and individuals in ways that undermine autonomy, presumption of innocence, and the very concept of freedom.

The Shift from Retrospective to Predictive Control

Traditional control systems respond to observed behavior: an individual acts, the action is observed, it is evaluated, and a response follows. This retrospective structure preserves a logical and temporal gap between behavior and consequence — consequence follows from action, action precedes consequence — that is foundational to most systems of individual accountability. You are held responsible for what you have done.

Predictive control breaks this retrospective structure by inserting a prediction model between observation and response. The system observes behavioral signals, constructs a prediction that certain future behavior is probable, and acts on that prediction — restricting access, intensifying monitoring, preemptively communicating sanctions, or blocking activity before the predicted behavior occurs. The individual is subject to control based not on their action but on the system's forecast of their action, which may or may not be accurate and which they have had no opportunity to disprove by not taking the predicted action.

This shift has several implications that generate the concerns associated with predictive control:

Irrefutability of prediction-based action: An individual who is preemptively restricted cannot demonstrate that they would not have violated the standard, because the preemption removed the opportunity. The prediction is both the basis for action and, by virtue of the action, impossible to test or challenge. This irrefutability makes predictive control particularly resistant to the normal mechanisms by which individuals can contest and reverse control decisions.

Error propagation at scale: Predictive models, however sophisticated, operate at population-level accuracy and produce false positive predictions — individuals identified as likely to violate who would not have violated — at rates that, when applied across large populations, generate large absolute numbers of incorrectly targeted individuals. Each false positive means an individual restricted, monitored, or penalized for a prediction rather than for an action.

Applications That Raise Predictive Control Concerns

Predictive control concerns arise across several application domains where algorithmic risk assessment is used to inform or determine consequential decisions:

Predictive policing uses algorithmic systems to forecast where crimes are likely to occur or which individuals are at elevated risk of offending, deploying police resources based on these predictions rather than solely on reported crime. Concerns center on the feedback loops produced: increased police presence in algorithmically flagged areas increases the probability of arrests in those areas, which generates more arrest data from those areas, which reinforces the algorithmic prediction, creating a self-fulfilling loop that amplifies rather than detects actual crime patterns.

Risk scoring in justice systems applies algorithmic risk assessment to individuals at points in the criminal justice process — bail determination, sentencing, parole decisions — producing scores that predict the likelihood of future offending. These scores typically incorporate variables like prior arrest history, employment, and residential stability that correlate with race and socioeconomic status, raising concerns that risk assessment perpetuates and legitimizes structural inequalities by encoding them in algorithmic predictions used to limit individual liberty.

Content and communication preemptive restriction on digital platforms involves systems that predict whether a user will violate community standards and restrict account functionality, require additional verification, or limit distribution before any violation occurs. These preemptive systems raise the irrefutability concern directly: a user preemptively restricted cannot demonstrate their innocence by not violating, because the restriction removed the opportunity.

The Autonomy Concern

The most fundamental concern with predictive control is its relationship to human autonomy — the capacity to make free choices and to be held responsible for those choices. Autonomy requires the freedom to make choices that have not yet been made: to face situations without preemptive determination of how one will respond. Predictive control systems, by acting on forecasts of future choices, substitute the algorithm's prediction for the individual's yet-unmade choice — treating predicted behavior as if it were actual behavior and closing off options before they are exercised.

When predictive control is applied to communication specifically — when the speech an individual has not yet uttered is predicted to violate standards and preemptively blocked — it restricts not actions but the capacity for action: the possibility of expressing views, making arguments, or communicating in ways the predictive system has classified as probable violations. This preemptive restriction of communicative possibility represents a particularly significant form of control because communication is the mechanism through which individuals exercise their capacity to contest, persuade, and participate in the public processes that determine the rules they live by.

Accountability and Transparency Requirements

Predictive control systems require accountability mechanisms that address their distinctive features. Because they act before harm occurs, they cannot be assessed against actual outcomes in the same way that retrospective control systems can; because they are probabilistic, their errors are distributed across populations rather than attributed to individual failures of judgment; because they are typically algorithmic, their operation is opaque to those they affect.

Accountability mechanisms appropriate to predictive control include: transparency about what behavioral variables are used in risk assessment, independent evaluation of model accuracy and fairness across demographic groups, due process rights for individuals to challenge predictive assessments before consequential actions are taken, and sunset provisions that require periodic reassessment of whether predictive models remain accurate and whether the population of individuals they affect is reasonable in scope.