"A few exciting words": information and entropy revisited | Lyn Robinson and David Bawden - Academia.edu
A review is presented of the relation between information and entropy, focusing on two main issues: the similarity of the formal definitions of physical entropy, according to statistical mechanics, and of information, according to information theory; and the possible subjectivity of entropy considered as missing information. The paper updates the 1983 analysis of Shaw and Davis. The difference in the interpretations of information given respectively by Shannon and by Wiener, significant for the information sciences, receives particular consideration. Analysis of a range of material, from literary theory to thermodynamics, is used to draw out the issues. Emphasis is placed on recourse to the original sources, and on direct quotation, to attempt to overcome some of the misunderstandings and oversimplifications that have occurred with these topics. While it is strongly related to entropy, information is neither identical with it, nor its opposite. Information is related to order and pattern, but also to disorder and randomness. The relations between information and the “interesting
complexity,” which embodies both patterns and randomness, are worthy of attention.
#informationtheory #entropy
Arieh Ben-Naim's book is out next week: Information, Entropy, Life and the Universe http:/
@NIMBios, I'd created a Storify timeline for #entropyWS in April if you want to include a link on its page https:/
The Homological Nature of Entropy
We propose that entropy is a universal co-homological class in a theory associated to a family of observable quantities and a family of probability distributions. Three cases are presented: (1) classical probabilities and random variables; (2) quantum probabilities and observable operators; (3) dynamic probabilities and observation trees. This gives rise to a new kind of topology for information processes, that accounts for the main information functions: entropy, mutual-informations at all orders, and Kullback–Leibler divergence and generalizes them in several ways. The article is divided into two parts, that can be read independently. In the first part, the introduction, we provide an overview of the results, some open questions, future results and lines of research, and discuss briefly the application to complex data. In the second part we give the complete definitions and proofs of the theorems A, C and E in the introduction, which show why entropy is the first homological invariant of a structure of information in four contexts: static classical or quantum probability, dynamics of classical or quantum strategies of observation of a finite system.
Shannon information; #homology theory; #entropy; #quantuminformation; homotopy of links; mutual informations; Kullback–Leiber divergence; trees; monads; partitions; #informationtheory
First, let's unpack your question a bit. What we're really doing is taking mathematical form, function, and theory and apply it in some sort of unifying manner to unify the areas of information theory and statistical thermodynamics.
You can also ask your question in the reverse direction to apply the mathematics of information to thermodynamics as well as there should be a mathematical duality of sorts, if in fact the two are directly related.
First, let's unpack your question a bit. What we're really doing is taking mathematical form, function, and theory and apply it in some sort of unifying manner to unify the areas of information theory and statistical mechanics and thermodynamics.
You can also ask your question in the reverse direction to apply the mathematics of information to thermodynamics as well as there should be a mathematical duality of sorts, if in fact the two are directly related.
The original link between the two areas stems from a paper by E.T. Jaynes entitled "Information Theory and Statistical Mechanics" (downloadable from http:/
Since then, in my opinion, the best source tying the fields together mathematically is A Farewell To Entropy: Statistical Thermodynamics Based on Information by Arieh Ben-Naim [http:/
As for other connections, I've not seen anything as direct, primarily as there aren't any other major concepts with the exact same name. However, because of the underlying mathematics, I strongly suspect that there are many similar 'concepts'. A few in particular, I have a feeling will come out of the more rigorous mathematical frameworks based on Lieb and Yngvason’s concptualization of thermodynamics (see Resource Convertibility (Part 1)[https:/
Another fruitful area of conceptual similarities is slowly grown out of quantum information theory and the statistical mechanics of black holes, though there is some heavily rarified air surrounding these given the required background needed to operate with them both simultaneously.
Response to Quora Question: Are there any popular science books on information theory?
There are a small handful that may meet your requirements:
Information, Entropy, Life and the Universe: What We Know and What We Do Not Know by Arieh Ben-Naim
http:/
An Introduction to Information Theory: Symbols, Signals and Noise by John R. Pierce
http:/
The Information: A History, A Theory, A Flood by James Gleick
http:/
Decoding the Universe: How the New Science of Information Is Explaining Everything in the Cosmos, from Our Brains to Black Holes by Charles Seife
http:/
Complexity: A Guided Tour by Melanie Mitchell
http:/
Programming the Universe: A Quantum Computer Scientist Takes on the Cosmos by Seth Lloyd
http:/
The Mathematical Theory of Communication by Claude E. Shannon and Warren Weaver
http:/
The Ben-Naim book is coming out at the end of May, and is probably the best popular science book on the topic that you're likely to find (I've read most of an early draft). Following that it's still very hard to beat Pierce's presentation all these years later. Seife wasn't too bad while Gleick comes at things from the most broad viewpoint and covers more history than anything else. Lloyd's book mentions it tangentially. Mitchell's book is excellent, and though it only has one chapter on information theory, some of the other areas it covers are tangential to the subject and very interesting in their own right.
Finally, after Ben-Naim and Robinson, I'd really recommend the UofI Press edition of Shannon/Weaver. Though you can get a copy of Shannon's paper for free online, the introduction by Warren Weaver is really excellent (and similar to Robinson's presentation) and then one can delve into a bit of Shannon's actual math, which isn't too difficult and should be able to be followed by most reasonable high school students. Shannon's text is the only one I've listed that has any mathematical equations, but the Weaver half of the book is certainly a popular science presentation without any math.
NIMBios is hosting a workshop on Information Theory and Entropy in Biological Systems this week (with streaming video) http:/
Perhaps you might also find further answers/information through the Mendeley Group for Information Theory and neuroscience: https:/
Your opening reminded me a bit of the motivating chapter of Andre Thess's book <a href="http://www.amazon.com/gp/product/3642133487/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=3642133487&linkCode=as2&tag=itbio-20&linkId=PVX5BVQGWFXLAGZZ">The Entropy Principle (Springer, 2011)</a> in which he uses the fable of Hans who manages to waste away a block of gold until he has nothing.
The book was an interesting textbook-like conceptualization of Lieb and Yngvason's work, which I always thought was due for a more thorough mathematical framework. I'm glad to see you not only reference them, but you appear to do what I've always wanted. Can't wait to delve further into your actual paper.