As readers of this blog will know, one of my enduring interests is how the concept of information appears in different domains. One aspect of this is the much-studied relation between information and the complex, and the multi-faceted, and arguably over-used, concept of entropy; see an older paper for background. Interest in this topic shows no sign of abating, and I mention here three interesting recent papers.
In a 2024 paper in Entropy, On the precise link between energy and information, Cameron Witkowski and co-workers from the University of Toronto revisit the classic analyses of the physics of information and entropy. Noting that “since the inception of thermodynamics, a delicate tension between physics and information has been unfolding”, they give a conceptual and in places strongly mathematical analysis, revisiting Maxell’s Demon, Szilard’s demonstration of the relation between entropy and information, and Landauer’s conception that ‘information is physical’. Their focus is on the nature of measurement in Szilard’s formulation, and on the energy cost of erasing information in Landauer’s examples. While not changing the findings of Szilard and Landauer – perpetual motion machines are indeed impossible, and processing information does take energy – their suggestion is that their approach gives a more general and satisfactory way of including information in physical theory.
In another Entropy paper from 2024, Steven Gimbel from Gettysburg College, gives a historical study of another of the founders of the entropy-information link, Ludwig Boltzmann. He examines influences on Boltzmann from contemporary scientific theories and controversies. The first of these was the modern approach to geology pioneered by James Hutton, whose focus on rock strata stemmed from the coal mining which powered the steam engines which provided the context for Carnot’s initial physical conception of entropy. The second was Charles Darwin’s theory of evolution of biological systems over time. Gimbel suggests that such theories encouraged Boltzmann’s statistical approach to the laws of thermodynamics, with its concepts of micro- and macro-states, and hence the link between entropy and our information about a system.
One of the more controversial, if not disreputable, uses of the concept of entropy is its application in the humanities and social sciences. An interesting example of a careful and scholarly use of entropy in this way is given by Herman Aksom of the University of Jyväskylä, in a 2022 article in International Journal of Organizational Analysis. This paper examines the idea of entropy as a factor in the erosion and decay of social institutions. Working within an information theoretic framework, Aksom identifies entropy at the level of information processing, corresponding with increasing uncertainty and the decrease of the value of information. Along the way, there is a critical discussion of the uses of entropy in the theory of institutions and organisations, associated with information processing, and with the opposite problems of a lack of information and of information overload. Tom Stonier and Norbert Wiener among others are invoked to support the idea that the same information (and hence entropy) concepts can used at physical, biological and social levels. This kind of rigorous analysis should serve to validate the use of the ideas of entropy, carefully related to information, in the social sciences as much as in the physical and biological realms.
Aksom quotes from the Czech writer Karel Capek’s novel Krakatit: “This is just [. . .] entropy, he said, thinking that this explained everything, and he repeated the strange word a few times.” There has perhaps been too much academic writing which overuses entropy uncritically, but studies such as these should help to clarify and demystify the information-entropy relation, and make it helpful in many contexts.
The Occasional Informationist 