4 Information and Signal Concepts

Information and signal concepts form the technical core of cybernetic communication theory's quantitative framework—the set of precisely defined ideas that allow communication to be analyzed with mathematical rigor rather than purely with verbal description. These concepts, originating primarily in Claude Shannon's mathematical theory of communication and Norbert Wiener's cybernetics, provide a formal vocabulary for describing what communication is, how it can fail, what its limits are, and how it can be optimized. Understanding them is essential for engaging seriously with the technical dimension of cybernetic communication theory.

The Distinction Between Signal and Information

The first and most fundamental conceptual distinction is between signal and information:

A signal is a physical variable—a quantity that changes over time and that can be transmitted through a physical medium. Electrical voltage, acoustic pressure, electromagnetic radiation intensity, and the orientation of magnetic particles on a recording medium are all physical signals. Signals carry information, but they are not identical to information: many different signals can carry the same information, and the same signal can carry different information depending on the code used to relate signals to meanings.

Information, in Shannon's technical sense, is the reduction of uncertainty produced by receiving a signal. Before a message is received, the recipient faces some degree of uncertainty about the state of the source; after the message is received, this uncertainty is reduced. The amount of information in a message is the amount by which it reduces the recipient's uncertainty.

This definition has several important implications:

• Information is relational, not absolute: the same signal carries different information to recipients with different prior uncertainty.
• Information is not content: two messages can carry identical information (identical uncertainty reduction) while having completely different semantic content.
• Completely predictable messages carry zero information: if a recipient knows with certainty what the next message will be, receiving it changes nothing.
• Completely surprising messages carry maximum information: if a recipient considers a message highly unlikely, receiving it produces large uncertainty reduction.

Shannon Entropy: Measuring Uncertainty and Information

The precise measure of information that Shannon introduced is the entropy formula. For a system that generates one of N possible symbols, each with probability p_i, the average information per symbol is:

This formula H is the Shannon entropy, measured in bits per symbol when the base-2 logarithm is used. Entropy is maximized when all symbols are equally probable (maximum uncertainty) and minimized when one symbol is certain (no uncertainty).

For a fair coin:

This means a fair coin flip generates exactly 1 bit of information—the amount needed to choose between two equally probable alternatives.

Entropy in this sense is formally identical to thermodynamic entropy in statistical mechanics, a connection Shannon recognized and that Wiener emphasized as fundamental: information is negative entropy (negentropy)—the measure of order that stands against the disorder measured by thermodynamic entropy.

Signal, Noise, and Signal-to-Noise Ratio

Every real communication channel introduces noise: random disturbances that alter the transmitted signal before it reaches the receiver. Noise is not merely unwanted signal; it is signal with no correlation to the intended message—pure randomness added to the information-carrying signal.

The signal-to-noise ratio (SNR) measures the relative strength of the intended signal versus the noise:

or more commonly expressed in decibels:

High SNR means the signal is much stronger than the noise, making accurate reception easier. Low SNR means the signal is comparable to or weaker than the noise, making accurate reception difficult.

In human communication, the SNR concept extends metaphorically: any factor that reduces the clarity or relevance of communicative content relative to the total communicative load—distracting environmental noise, excessive hedging, irrelevant content, emotional interference—reduces the effective SNR of the communication.

Channel Capacity: The Shannon-Hartley Theorem

The Shannon-Hartley theorem establishes the maximum rate at which information can be reliably transmitted through a channel with a specified bandwidth and signal-to-noise ratio:

Where:

• C is the channel capacity in bits per second.
• B is the channel bandwidth in Hertz (the range of frequencies the channel can transmit).
• S/N is the signal-to-noise ratio.

This theorem establishes a hard upper limit: no matter how cleverly the information is encoded, more than C bits per second cannot be transmitted reliably through a channel with bandwidth B and SNR S/N. Shannon's coding theorem further establishes that codes exist that approach this limit arbitrarily closely through sufficiently long code blocks.

The channel capacity concept is one of the most practically consequential results in the history of communication engineering: it tells engineers exactly what is achievable and what is not, enabling efficient system design without wasted effort on impossible goals.

Redundancy and Error Correction

A signal is redundant to the degree that it contains more structure than the minimum needed to convey its information. In a message where any symbol can be any value with equal probability, there is no redundancy: every symbol carries the maximum possible information. In a message where the value of each symbol is highly predictable from the preceding symbols (as in natural language, where "TH" is almost always followed by a vowel), redundancy is high.

Shannon's fundamental theorem of channel coding establishes that reliable communication over a noisy channel is always possible, at any rate below channel capacity, if the information is encoded with sufficient redundancy. The redundant encoding distributes the information across many symbols in a way that allows the receiver to recover the original message even if some symbols are altered by noise.

The mathematically precise form of an error-correcting code specifies:

• A code rate r (the fraction of transmitted symbols that carry actual information; the rest are redundancy).
• A minimum Hamming distance d (the minimum number of symbols that must be altered to transform one valid codeword into another valid codeword).
• An error-correcting capacity (the maximum number of symbol errors per codeword that the code can correct, equal to (d-1)/2).

In human communication, redundancy appears in natural language (grammar, context, shared knowledge all allow damaged utterances to be repaired), in conversational repair sequences, and in organizational communication protocols that require multiple confirmation steps for important transactions.

Mutual Information

Mutual information measures how much knowing the receiver's output Y reduces uncertainty about the sender's input X:

where H(X|Y) is the conditional entropy of X given Y—the remaining uncertainty about X after Y is known. Mutual information is symmetric: I(X;Y) = I(Y;X), and equals the channel capacity C when the input distribution is chosen optimally.

Mutual information provides a natural measure of communication effectiveness: the closer the mutual information between sender and receiver to the entropy of the source, the more effectively the channel is communicating. In human communication, mutual information provides a way to quantify how well interlocutors understand each other: high mutual information means the listener's state accurately reflects the speaker's state; low mutual information means much uncertainty remains about the speaker's intended message.

The Bandwidth-Resolution Trade-off

An important practical consequence of the signal and information framework is the bandwidth-resolution trade-off: more rapid communication requires wider bandwidth but allows less precision per unit time, while more precise communication requires narrower bandwidth but is slower.

In human terms: a brief message can convey the gist of a complex situation (high bandwidth, low resolution); a detailed report can convey precise nuance but takes much longer (low bandwidth, high resolution). Every communication system must negotiate this trade-off based on the available time and the required precision.

Information Concepts Beyond Shannon

Shannon's syntactic information theory measures quantities—how much information—but says nothing about what the information is about, whether it is true, or what it means. Three extensions of information theory address these limitations in different ways:

Semantic information theory attempts to measure the meaning-relevance of information, not just its uncertainty-reduction. Various proposals have been made for semantic information measures, but none has achieved the mathematical elegance and universal applicability of Shannon's syntactic theory.

Pragmatic information theory addresses the use of information—what the receiver does with it. The same information can lead to very different actions in different contexts; pragmatic theories attempt to account for how information drives behavior.

Ecological information theory (James Gibson's framework) focuses on information available in the environment—the structured patterns that allow organisms to perceive affordances directly without internal computation. Gibson's concept of information is fundamentally different from Shannon's: it is not a reduction of uncertainty but a direct specification of environmental structure.

These extensions reflect the recognition that Shannon's framework, powerful as it is, captures only one dimension of communication—the syntactic, statistical dimension—while human communication additionally involves semantic, pragmatic, and ecological dimensions that require supplementary frameworks.