Paper Summary

Citation: Shannon, C. E. (1949). Communication in the Presence of Noise. Proceedings of the IRE, 37(1), 10-21. https://doi.org/10.1109/JRPROC.1949.232969

Publication: Proceedings of the IRE, 1949

What kind of paper is this?

This is a foundational “big idea” and theoretical paper. It establishes the mathematical framework for modern information theory and defines the ultimate physical limits of communication for an entire system, from the information source to the final destination.

What is the motivation?

The central motivation was to develop a general theory of communication that could quantify information and determine the maximum rate at which it can be transmitted reliably over a noisy channel. Prior to this work, communication system design was largely empirical. Shannon sought to create a mathematical foundation to understand the trade-offs between key parameters like bandwidth, power, and noise, independent of any specific hardware or modulation scheme.

What is the novelty here?

The novelty is a complete, end-to-end mathematical theory of communication built upon several groundbreaking concepts and theorems:

  1. Geometric Representation of Signals: Shannon introduced the idea of representing signals as points in a high-dimensional vector space. A signal of duration $T$ and bandwidth $W$ is uniquely specified by $2TW$ numbers (its samples), which are treated as coordinates in a $2TW$-dimensional space. This transformed problems in communication into problems of high-dimensional geometry. In this representation, signal energy corresponds to squared distance from the origin, and noise introduces a “sphere of uncertainty” around each transmitted point.
Sphere packing illustration showing the geometric interpretation of channel capacity. A large dashed circle represents the total signal space with radius proportional to the square root of P+N. Inside are multiple smaller blue circles (uncertainty spheres) with radius proportional to the square root of N, each centered on a distinct message point.
Sphere Packing and Channel Capacity: Each transmitted message corresponds to a point in high-dimensional signal space. Noise creates an ‘uncertainty sphere’ of radius $\sqrt{N}$ around each point. The channel capacity equals how many non-overlapping uncertainty spheres can be packed into the total signal sphere of radius $\sqrt{P+N}$.

  1. Theorem 1 (The Sampling Theorem): The paper provides an explicit statement and proof that a signal containing no frequencies higher than $W$ is perfectly determined by its samples taken at a rate of $2W$ samples per second (i.e., spaced $1/2W$ seconds apart). This theorem is the theoretical bedrock of all modern digital signal processing.

  2. Theorem 2 (Channel Capacity for AWGN): This is the paper’s most celebrated result, the Shannon-Hartley theorem. It provides an exact formula for the capacity $C$ (the maximum rate of error-free communication) of a channel with bandwidth $W$, signal power $P$, and additive white Gaussian noise of power $N$: $$ C = W \log_2 \left(1 + \frac{P}{N}\right) $$ It proves that for any transmission rate below $C$, a coding scheme exists that can achieve an arbitrarily low error frequency.

Two plots illustrating Shannon's key theorems: (Left) The ideal capacity curve showing bits per cycle vs SNR in dB, with an example operating point at 10dB. (Right) The sampling theorem demonstrating how a continuous signal is perfectly captured by samples taken at the Nyquist rate of 2W.
Left: Shannon’s ideal capacity curve, showing how channel capacity (in bits per cycle) increases logarithmically with signal-to-noise ratio. Right: The sampling theorem in action, where a band-limited continuous signal is fully determined by discrete samples taken at twice its maximum frequency.
  1. Theorem 3 (Channel Capacity for Arbitrary Noise): Shannon generalized the capacity concept to channels with any type of noise, not just white Gaussian noise. He showed that the capacity for a channel with arbitrary noise of power $N$ is bounded by the noise’s entropy power $N_1$ (a measure of its randomness). This demonstrated the robustness and generality of the capacity concept.
The water-filling principle for optimal power allocation. Gray area shows noise power N(f) varying across frequencies, orange area shows signal power P(f) allocated to fill up to a constant level lambda, such that P(f) + N(f) equals a constant.
The Water-Filling Principle: When noise power varies across frequencies, optimal capacity is achieved by allocating more signal power to ‘quieter’ frequency bands. Like filling a container with water, power is poured in until the total (signal + noise) reaches a constant level $\lambda$. Frequencies with noise above this threshold receive no power at all.

  1. Theorem 4 (Source Coding Theorem): This theorem addresses the information source itself. It proves that it’s possible to encode messages from a discrete source into binary digits such that the average number of bits per source symbol approaches the source’s entropy, $H$. This establishes entropy as the fundamental limit of data compression.

  2. Theorem 5 (Information Rate for Continuous Sources): For continuous (analog) signals, Shannon introduced a concept foundational to rate-distortion theory. He defined the rate $R$ at which a continuous source generates information relative to a specific fidelity criterion (i.e., a tolerable amount of error, $N_1$, in the reproduction). This provides the basis for all modern lossy compression algorithms.

What experiments were performed?

The paper is primarily theoretical, with “experiments” consisting of rigorous mathematical derivations and proofs. The channel capacity theorem, for instance, is proven using a geometric sphere-packing argument in the high-dimensional signal space.

However, Shannon does include a quantitative comparison against existing 1949 technology. He plots his theoretical “Ideal Curve” against Pulse Code Modulation (PCM) and Pulse Position Modulation (PPM) systems in Figure 6. This comparison reveals that contemporary PCM systems operated approximately 8 dB below the ideal power limit, providing engineers with a concrete measure of how much room for improvement remained.

What were the outcomes and conclusions drawn?

The primary outcome was a complete, unified theory that quantifies both information itself (entropy) and the ability of a channel to transmit it (capacity).

  • Decoupling of Source and Channel: A key conclusion is that the problem of communication can be split into two distinct parts: source coding (compressing the message to its entropy rate, $H$) and channel coding (adding structured redundancy to protect against noise). A source can be transmitted reliably if and only if its rate $R$ (or entropy $H$) is less than the channel capacity $C$.

  • The Limit is on Rate, Not Reliability: Shannon’s most profound conclusion was that noise in a channel does not create an unavoidable minimum error rate; rather, it imposes a maximum rate of transmission. Below this rate, error-free communication is theoretically possible.

  • The Threshold Effect: To approach capacity, one must map a lower-dimensional message space into the high-dimensional signal space efficiently, like winding a “ball of yarn” to fill the available signal sphere. This complex mapping creates a sharp threshold effect: below a certain noise level, recovery is essentially perfect; above it, the system fails catastrophically because the “uncertainty spheres” around signal points begin to overlap. This explains the “cliff” behavior seen in digital communication systems, where performance is excellent until it suddenly isn’t.

  • The Exchange Relation: Shannon explicitly states that the key parameters $T$ (time), $W$ (bandwidth), $P$ (power), and $N$ (noise) can be “altered at will” as long as the channel capacity $C$ remains constant. This exchangeability is a fundamental insight for system architects, enabling trade-offs such as using more bandwidth to compensate for lower power.

  • Characteristics of an Ideal System: The theory implies that to approach the channel capacity limit, one must use very complex and long codes. An ideal system exhibits four key properties: (1) the transmission rate approaches $C$, (2) the error probability approaches zero, (3) the transmitted signal’s statistical properties approach those of white noise, and (4) the required delay increases indefinitely. This final constraint is a crucial practical limitation: achieving near-capacity performance requires encoding over increasingly long message blocks, introducing latency that may be unacceptable for real-time applications.