4.5 Channel Capacity

Channel capacity is the maximum rate at which information can be transmitted reliably over a communication channel. It represents a fundamental limit: no matter how sophisticated the encoding and decoding schemes are, it is impossible to communicate information faster than the channel capacity without incurring an irreducible probability of error. Conversely, Shannon's channel coding theorem guarantees that for any rate below the channel capacity, there exist encoding and decoding schemes that achieve arbitrarily low error probability.

The concept was formalized by Claude Shannon in 1948 as part of his foundational work in information theory. Shannon defined channel capacity in terms of mutual information, which measures the statistical dependence between the input and output of a channel. The capacity is the maximum mutual information achievable by optimizing over all possible input distributions:

where X is the channel input random variable, Y is the channel output random variable, and I(X; Y) denotes the mutual information between them. The optimization searches over all probability distributions on the input alphabet to find the one that maximizes the information transmitted per channel use.

For the most widely studied model, the additive white Gaussian noise (AWGN) channel, Shannon derived a closed-form expression for capacity. In this model, the received signal equals the transmitted signal plus a Gaussian noise term. The capacity in bits per second is given by:

where B is the bandwidth of the channel in hertz, S is the average signal power, and N is the noise power within that bandwidth. This formula, commonly called the Shannon–Hartley theorem, reveals that capacity grows logarithmically with the signal-to-noise ratio and linearly with the bandwidth.

The two main parameters in the Shannon–Hartley theorem, bandwidth and signal-to-noise ratio, have different practical implications. Bandwidth is often a scarce and regulated resource, particularly in wireless communication, where spectrum is allocated by regulatory authorities. Signal power can in principle be increased, but logarithmic scaling means that doubling the signal-to-noise ratio yields only a constant additive increase in capacity, making large power increases progressively less effective.

Channel capacity applies to memoryless channels, in which the noise affecting each transmitted symbol is independent of previous symbols. For channels with memory, where the noise process is correlated across time, the capacity concept extends to a rate defined by the limit of mutual information per channel use as the blocklength grows. Computing capacity for channels with memory is generally more complex and depends on the specific statistical structure of the noise process.

Discrete channels, in which both the input and output alphabets are finite, have their capacity determined by the channel transition matrix. Each entry of this matrix specifies the conditional probability of receiving a particular output symbol given a particular input symbol. For a binary symmetric channel, where each bit is flipped independently with probability p, the capacity is:

where H(p) is the binary entropy function:

When p equals 0 or 1, the channel is noiseless or perfectly invertible, giving capacity of 1 bit per channel use. When p equals 0.5, each output bit is completely independent of the input, and capacity drops to zero, meaning no information can be transmitted reliably.

In cybernetic communication theory, channel capacity plays a central role in understanding the fundamental limits of control and information transfer in systems. A feedback control system can only be as responsive as its communication links allow, and any control information flowing back from the environment to the controller must respect the capacity limits of the signaling pathways. This connection between channel capacity and the effectiveness of feedback loops links information theory directly to the cybernetic study of self-regulating systems.

The practical significance of channel capacity extends to all modern communication technologies. Engineers designing wireless networks, fiber-optic systems, satellite links, and digital storage media all work under the constraint of approaching their respective channel capacities as closely as possible, driving the development of sophisticated coding schemes such as turbo codes, polar codes, and LDPC codes that achieve transmission rates close to the theoretical limit.