The great Hopfield network debate
AI Histories #3: A bitter clash about machine learning's finest hour
I’ve recently opened reader pledges with a goal of reaching 100 by May 31. So far we’ve on track, but I need your help to get there. All you have to do is pledge $5, so that I know I have your support for the moment I turn on paid subscriptions (and eventually try to make this work full time). Thank you to everyone who was kindly pledged so far. It means a lot.
John Hopfield won the Nobel Prize for physics last year. Alongside Geoffrey Hinton, Hopfield was recognised for foundational contributions to machine learning that would ultimately make the likes of ChatGPT and Gemini possible. His great invention was the network that bears his name, whose development is generally considered to be an essential moment in the machine learning canon. Speaking to the awarding committee, he discussed how he first began to approach work on the systems:
You don’t leap into a problem overall saying, I want to understand how mind works. You have to build up from the bottom. If you were doing weather, you would say, well, I want to understand what storms are without going back to interacting air nitrogen molecules.
Analogies, in other words, are key. Hopfield’s 1982 paper is commonly credited with the invention of the ‘Hopfield network’, a type of recurrent neural network where all units are connected to each other. Typically used for pattern recognition and memory storage, the emergence of the the systems seemed to show that there was still life in the ‘connectionist’ branch of AI (the ancestor of deep learning). As researcher Tom Schwartz put it ‘Hopfield should be known as the fellow who brought neural nets back from the dead.’
Hopfield is no doubt a hugely important figure, but wasn’t the first to design and implement a fully connected network. Stanford researcher William Little had previously introduced versions of the networks in 1976, and so had Japanese neuroscientist Shun'ichi Amari in 1972.
The cognitive scientist Stephen Grossberg went further, arguing that he built the specific architecture that Hopfield made famous. As Grossberg, who first wrote about the ideas described in the Hopfield model in 1957, put it: ‘I don’t believe that this model should be named after Hopfield. He simply didn’t invent it. I did it when it was really a radical thing to do.’
But as every scientist knows, research needs rhetoric and papers need presentation. Hopfield removed chunks of mathematical descriptions in favour of compelling prose written for cognitive scientists. He published his paper in the influential Proceedings of the National Academy of Science and travelled extensively to talk about ‘his’ networks. These are the primary reasons that today we know the systems not as Grossberg networks, but as Hopfield networks.
One of the paper’s most influential ideas involved conceptualising the networks as ‘spin glasses’, a term derived from the magnetic state of matter. ‘Spin’ is a quantum property of particles that allows them to behave as miniscule magnets that can point in different directions. ‘Glass’ draws from an analogy with conventional glass, which is known for its irregular, amorphous structure at the atomic level. Atoms are arranged in a disordered state in a typical glassy material, unlike in crystal forms where atoms have a more regular arrangement.
In the Hopfield network, the system’s dynamics—how these states settle into patterns—are inspired by the way magnetic spins interact in physical systems. Hopfield’s work connected this idea, the potential for systems to transition from a disordered state to a stable one, to the concept of ‘associative memory’. Based loosely on the brain, the associative memory concept holds that if you give a system a piece of the desired output it can ‘remember’ the rest.
The connection was straightforward: just as a spin glass system transitions from a high energy state to a low energy state, a Hopfield network minimises its energy to represent and retrieve stored patterns. It was this analogy, which explained how a sea of small parts could settle into stable states representing stored memories, that encouraged the next generation of researchers to couple the operation of Hopfield networks with principles similar to those observed in physical systems.
Jack Cowen, an influential researcher who worked with Hopfield, understood the symbolic currency of the idea. He said ‘I think that's neat stuff, but I still think it's an artificial system, as is the antisymmetric one. It may have nothing to do with the way things really work in the nervous system, but it's a very interesting idea.’
By drawing a parallel between associative memory and the behaviour of physical systems like spin glasses, Hopfield reinforced the idea that complex cognitive processes could emerge from the collective behaviour of simple interacting units. This perspective supported the well-established notion that the brain’s higher-order cognitive functions could be explained through the interactions of simpler components.
As Hopfield explained: ‘Computational properties of use to biological organisms or to the construction of computers can emerge as collective properties of systems having a large number of simple equivalent components (or neurons).’ The result, according to this model, is that intelligence is an emergent property that may be produced through the interaction of smaller, simpler units. Faced with such a conclusion, it is perhaps unsurprising that the paper caught fire.
Toy models
To explain the functioning of the systems, Hopfield leaned heavily on the famous law coined by Canadian psychologist Donald Hebb: ‘When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased.’ You might know it as ‘neurons that fire together, wire together.’
Of course, Hopfield networks are not brains. And neither are they spin glasses. These types of comparisons are better thought of as ‘toy models’ that seek to help scientists understand the world. Frank Rosenblatt, on whose famous perceptron algorithm Hopfield aimed to build, stressed the importance of toy models to AI research in 1961:
‘The model is a simplified theoretical system, which purports to represent the laws and relationships which hold in the real physical universe…the model deliberately neglects certain complicating features of the natural phenomena under consideration, in order to obtain a more readily analyzed system, which will suggest basic principles that might be missed among the complexities of a more accurate representation.’
These models tell researchers about the problem they are trying to solve, but they also widen the epistemological field to accommodate new perspectives. Philosophers of science Knuttila and Loettgers, for example, have shown that mental models ‘provide modelers with a powerful cognitive strategy to transfer concepts, formal structures, and methods from one discipline to another’.
Through this process of analogical reasoning, the twin abstractions of spin glass modelling and neurophysiology offered a way of describing the functioning of the systems that was both persuasive and epistemically valuable. It may have won Hopfield a Nobel Prize, but that would have been small comfort to Stephen Grossberg.