4.7 Information Difference

Information difference refers to the disparity between two states of knowledge, representations, or probability distributions, capturing how much one information state diverges from another. In cybernetic communication theory, information difference underlies the meaningful content of a signal: a signal carries information precisely because it creates a difference, altering the receiver's knowledge state in a way that distinguishes between possible alternatives. Gregory Bateson's frequently cited formulation encapsulates this principle: information is "a difference that makes a difference," meaning that only those distinctions which alter the behavior or state of a system constitute genuine information.

At the most fundamental level, an information difference exists between a prior state of knowledge and a posterior state of knowledge after receiving a message. Before a signal arrives, a receiver holds a prior probability distribution over possible states of the world. After the signal, the distribution shifts to a posterior. The information difference is the change induced by this update. If the signal changes the distribution significantly, it carries a large information difference; if it barely shifts the probabilities, it carries a small one.

This concept connects directly to the Kullback-Leibler divergence, which is the standard information-theoretic measure of the difference between two probability distributions P and Q:

The KL divergence is always non-negative and equals zero only when P and Q are identical distributions. It can be interpreted as the expected information gain when updating from distribution Q to distribution P, or equivalently, the inefficiency incurred by assuming the distribution is Q when it is actually P. In the context of information difference, the KL divergence from prior to posterior measures the total information conveyed by the observation.

Information difference is also central to mutual information, which quantifies the information difference created in one variable by knowing the value of another. The mutual information between two random variables X and Y measures how much knowledge of Y reduces uncertainty about X:

This expression is itself an information difference: the difference between the entropy of X without knowledge of Y, and the conditional entropy of X given Y. When Y provides complete information about X, the mutual information equals the full entropy of X and the conditional entropy drops to zero. When Y and X are independent, mutual information is zero, reflecting the absence of any information difference.

In the Batesonian cybernetic tradition, information difference is not merely a statistical concept but an ontological one. For a system to respond to information, it must be sensitive to differences in its environment. A thermostat detects a temperature difference between the current state and a set point; a neuron fires when the difference in electrochemical potential across its membrane exceeds a threshold; a controller adjusts its output when the difference between measured and desired states falls outside acceptable bounds. In each case, the system's functioning depends on its capacity to detect, amplify, and respond to information differences.

The concept of information difference also illuminates the significance of thresholds in communication and perception. Not every difference is detectable or meaningful. There is a minimum information difference that a system can reliably distinguish, analogous to the just-noticeable difference studied in psychophysics. Below this threshold, differences in the incoming signal are treated as noise and produce no response. This perceptual limit means that the effective information difference perceived by a receiver depends not only on the objective statistical difference between signals but also on the sensitivity and resolution of the receiving system.

Across scales of analysis, information difference appears in the study of how complex systems maintain their organization. Differences between the current state of a system and a target state generate corrective signals through feedback loops. In this way, information differences drive the error-correction and homeostatic processes that cybernetics aims to understand. The reduction of information differences through feedback is the operational mechanism by which systems achieve and maintain order against the tendency toward disorder described by the second law of thermodynamics.