Information theory

Information theory is the study of how much information a message carries and how reliably it can be sent. Its central quantity is entropy: a message is informative to the degree that it is surprising, and Shannon measured that surprise in bits. Content that you could have predicted — a redundant letter, a repeated word — carries little information; content you could not predict carries a lot. From this one idea Shannon derived hard limits: the smallest number of bits a source can be compressed to, and the greatest rate at which those bits can be pushed through a noisy channel without error. Crucially, information theory ignores meaning entirely — it counts only the statistics of the symbols, which is exactly why it applies equally to speech, images, DNA, and machine code.

Before 1948, engineers built communication systems by intuition and rule of thumb, with no way to say what was even possible. Shannon gave the field its physics: exact, provable bounds on compression and on error-free transmission, and the shocking result that noise sets a *ceiling on rate, not on accuracy* — below capacity, errors can be driven arbitrarily low. That reframed communication and storage as coding problems with known optimal answers, and it supplied the common currency, the bit, in which all information could be measured and traded regardless of its form.

Information theory is the shared root of data compression (ZIP, JPEG, MP3), error correction (from Voyager's signals to every hard drive and Wi-Fi link), cryptography, and the modern understanding of channel capacity that governs how fast any wire, fibre, or radio band can carry data. It gave computing and communication a unified quantitative foundation — the bit as a universal unit — and its concept of entropy reaches into physics, statistics, biology, and machine learning. Together with Boolean logic and the stored-program computer, it completes the theoretical trio underneath the digital age.