4.8 Information Reduction of Uncertainty

Information as the reduction of uncertainty is one of the foundational principles of both information theory and cybernetic communication theory. In this framework, information is not defined by its meaning, truth value, or semantic content, but by the degree to which it resolves ambiguity about the state of the world. A receiver facing multiple possible alternatives has uncertainty about which alternative is actual. When a signal arrives and eliminates some of those alternatives or shifts their probabilities, the receiver's uncertainty decreases. That decrease in uncertainty is precisely the information received.

This perspective links information directly to the concept of entropy from statistical mechanics, which Shannon adapted for communication theory. Before receiving any signal about a random variable X, the receiver's uncertainty about X is quantified by Shannon entropy:

After receiving an observation Y, the residual uncertainty about X is the conditional entropy of X given Y:

The reduction in uncertainty brought about by the observation is the mutual information:

Mutual information is always non-negative, reflecting the data processing inequality: observing additional data cannot on average increase uncertainty. The more an observation constrains the possible values of X, the greater the uncertainty reduction and thus the greater the information conveyed.

The relationship between uncertainty and information is symmetric: maximum uncertainty corresponds to maximum potential information gain. A completely predictable event, one with probability 1, carries zero uncertainty and thus provides zero information when it occurs. Conversely, a uniform distribution over many equally likely outcomes carries maximum entropy and can yield the greatest possible information upon resolution. This explains why unpredictable sources, such as random processes, are also information-rich sources in the technical sense.

In the cybernetic tradition developed by Norbert Wiener, the reduction of uncertainty through information plays a central role in understanding how purposive systems function. A control system cannot steer toward a goal if it lacks information about the current state of the system. Each sensor reading reduces the controller's uncertainty about the plant's position, speed, or other relevant variables, enabling corrective action. The quality of control is thus bounded by the rate at which information reduces uncertainty, which is in turn bounded by the channel capacity of the sensor-to-controller communication path.

The concept also connects to Bayesian inference, where prior uncertainty about a hypothesis is represented as a prior probability distribution, and evidence updates that distribution to a posterior. The reduction in entropy from prior to posterior, sometimes called the information gain or relative entropy, measures the uncertainty-reducing impact of the evidence. This Bayesian interpretation extends the concept beyond communication channels to all forms of observation and learning, making information as uncertainty reduction a general epistemological principle.

In practical communication systems, the goal of reducing uncertainty as efficiently as possible motivates the design of error-correcting codes and efficient modulation schemes. An unreliable channel leaves residual uncertainty at the receiver even after the signal arrives, measured by the conditional entropy H(X|Y). By adding redundancy and using sophisticated decoding, engineers reduce this residual uncertainty, allowing the decoded message to approach the transmitted message with high confidence. The limiting case of zero residual uncertainty corresponds to error-free communication, achievable below the channel capacity by Shannon's fundamental theorem.

Uncertainty reduction as a definition of information also has social and organizational implications recognized in cybernetic communication theory. Organizations and decision-making systems are understood as reducers of uncertainty: they gather data, process it, and arrive at states of reduced ambiguity that permit coordinated action. The capacity of an organization to reduce uncertainty about its environment, competitors, resources, and internal state determines its adaptive capability, making information management central to organizational effectiveness.