My primary interest is in brain function. Because the principal job of
the brain is to process information, it is necessary to define exactly
what information is. For that, there is no substitute for Claude
Shannon’s theory of information. This theory is not only quite
remarkable in its own right, but it is essential for telecoms,
computers, machine learning (and understanding brain function).
I have written ten "tutorial introduction" books, on topics which vary
from quantum mechanics to AI.
In a parallel universe, I am still an Associate Professor at the
University of Sheffield, England.
Originally developed by Claude Shannon in the 1940s, information theory laid the foundations for the digital revolution, and is now an essential tool in telecommunications, genetics, linguistics, brain sciences, and deep space communication. In this richly illustrated book, accessible examples are used to introduce information theory in terms of everyday games like ‘20 questions’ before more advanced topics are explored. Online MatLab and Python computer programs provide hands-on experience of information theory in action, and PowerPoint slides give support for teaching.
Written in an informal style, with a comprehensive glossary and tutorial appendices, this text is an ideal primer for novices who wish to learn the essential principles and applications of information theory.
Pierce was a contemporary of Claude Shannon (inventor of information theory), so he learned information theory shortly after it was published in 1949. Pierce writes in an informal style, but does not flinch from presenting the fundamental theorems of information theory. Some would say his style is too wordy, and the ratio of words/equations is certainly very high. Nevertheless, this book provides a solid introduction to information theory. It was originally published in 1961, so it is a little dated in terms of topics covered. However, because it was re-published by Dover in 1981, it is also fairly cheap. Overall, this is a sensible first book to read on information theory.
"Uncommonly good...the most satisfying discussion to be found." — Scientific American. Behind the familiar surfaces of the telephone, radio, and television lies a sophisticated and intriguing body of knowledge known as information theory. This is the theory that has permitted the rapid development of all sorts of communication, from color television to the clear transmission of photographs from the vicinity of Jupiter. Even more revolutionary progress is expected in the future. To give a solid introduction to this burgeoning field, J. R. Pierce has revised his well-received 1961 study of information theory for a second edition. Beginning with the origins…
This is a more comprehensive and mathematically rigorous book than Pierce’s book. For the novice, it should be read-only after first reading Pierce’s more informal text. Due to its vintage, the layout is fairly cramped, but the content is impeccable. At almost 500 pages, it covers a huge amount of material. This was my main reference book on information theory for many years, but it now sits alongside more recent texts, like MacKay’s book (see below). It is also published by Dover, so it is reasonably priced.
Written for an engineering audience, this book has a threefold purpose: (1) to present elements of modern probability theory — discrete, continuous, and stochastic; (2) to present elements of information theory with emphasis on its basic roots in probability theory; and (3) to present elements of coding theory. The emphasis throughout the book is on such basic concepts as sets, the probability measure associated with sets, sample space, random variables, information measure, and capacity. These concepts proceed from set theory to probability theory and then to information and coding theories. No formal prerequisites are required other than the usual undergraduate…
This is the modern standard text on information theory. It is both comprehensive and highly technical. The layout is spacey, and the authors make good use of the occasional diagram to explain geometric aspects of information theory. One feature I really like is the set of historical notes and a summary of equations at the end of each chapter.
The latest edition of this classic is updated with new problem sets and material The Second Edition of this fundamental textbook maintains the book's tradition of clear, thought-provoking instruction. Readers are provided once again with an instructive mix of mathematics, physics, statistics, and information theory. All the essential topics in information theory are covered in detail, including entropy, data compression, channel capacity, rate distortion, network information theory, and hypothesis testing. The authors provide readers with a solid understanding of the underlying theory and applications. Problem sets and a telegraphic summary at the end of each chapter further assist readers. The…
This is considered to be a modern classic on information theory. It is a fairly readable text that roams far and wide over many topics. MacKay was extraordinarily clever, curious, and generous (which is why he made the book freely available). His wide range of interests included coding theory and artificial neural networks (machine learning), which occupy a few chapters of the book. In describing the difference between Bayesian and frequentist statistical methods, MacKay pulls no punches, and he can be heard laughing through gritted teeth as he tries to make sense of the nonsensical. This is one of those books in which the author’s many throw-away asides are as enlightening as the main text. The book’s website (below) also has a link to an excellent series of video lectures by MacKay. It is also available for free, just use the direct link below.
Information theory and inference, taught together in this exciting textbook, lie at the heart of many important areas of modern technology - communication, signal processing, data mining, machine learning, pattern recognition, computational neuroscience, bioinformatics and cryptography. The book introduces theory in tandem with applications. Information theory is taught alongside practical communication systems such as arithmetic coding for data compression and sparse-graph codes for error-correction. Inference techniques, including message-passing algorithms, Monte Carlo methods and variational approximations, are developed alongside applications to clustering, convolutional codes, independent component analysis, and neural networks. Uniquely, the book covers state-of-the-art error-correcting codes, including low-density-parity-check codes, turbo…
This is really two books, strapped together. The first book by Weaver is an informal introduction to the ideas implicit in the second book by Shannon. Shannon’s book is naturally quite dated in its use of language (e.g. uncertainty is pronounced equivocation), and in its references to ‘current’ technology (e.g. PCM). Despite these caveats, it is still a surprisingly accessible book. Finally, because information theory was developed almost exclusively by Claude Shannon, reading the theory explained in his own words gives some insight into how on Earth he managed to come up with such a radical set of ideas.
Scientific knowledge grows at a phenomenal pace--but few books have had as lasting an impact or played as important a role in our modern world as The Mathematical Theory of Communication, published originally as a paper on communication theory more than fifty years ago. Republished in book form shortly thereafter, it has since gone through four hardcover and sixteen paperback printings. It is a revolutionary work, astounding in its foresight and contemporaneity. The University of Illinois Press is pleased and honored to issue this commemorative reprinting of a classic.
I am a highly experienced outdoorsman, musician, songwriter, and backcountry guide who chose teaching as a day job. As a writer, however, I am a promoter of creative and literary nonfiction, especially nonfiction that features a thematic thread, whether it be philosophical, conservation, historical, or even unique experiential. The thread I used for thirty years of teaching high school and honors English was the thread of Conservation, as exemplified by authors like Aldo Leopold, Rachel Carson, Edward O. Wilson, Al Gore, Henry David Thoreau, as well as many other more contemporary authors.
I have woven numerous delightful and descriptive true life stories, many from my adventures as an outdoorsman and singer songwriter, into my life as a high school English teacher. I think you'll find this work both entertaining as well as informative, and I hope you enjoy the often lighthearted repartee and dialogue that enhances the stories and experiences.
When I started teaching in the early 1990s, I brought into the classroom with me my passions for nature, folk music, and creativity. This book holds something new and engaging with every chapter and can be enjoyed by all sorts of readers, particularly those who enjoy nonfiction that employs wit, wisdom, humor, and even some down-to-earth philosophy.
Creativity, Teaching, and Natural Inspiration follows the evolution of a high school English teacher as he develops a creative and innovative teaching style despite being juxtaposed against a public education system bent on didactic, normalizing regulations and political demands. Doherty crafts an engaging nonfiction story that utilizes memoir, anecdote, poetry, and dialogue to explore how mixing creativity and pedagogy can change the way budding students visualize creative writing: A chunk of firewood plunked on a classroom table becomes part of a sawmill, a mine timber, an Anasazi artifact...it also becomes a poem, a song, an essay, and a memoir. The…
Interested in
information theory,
machine learning,
and
python?
10,000+ authors have recommended their favorite books and what they love about them.
Browse their picks for the best books about
information theory,
machine learning,
and
python.
