Humanoid robots are cool. Like driverless cars, they are one of those rare pieces of modern technology that feel appropriately futuristic. Should the makers be able to judo flip teleoperation into full automation, we can expect the stuff of sci-fi serials to be used enthusiastically by anyone who can get their hands on them.

This week, X1 made its NEO model available for pre-order. For $500 a month and a deposit, you can get one of these 5’6 guys delivered to your house in 2026 (so long as you live in the United States). By all accounts NEO, which looks a bit like a walking 2000s PC speaker, can do a pretty good job at helping you around the house. X1 CEO Bernt Øyvind Børnich called it “robotics slop” insofar as the robot (or for now, its human pilot) can perform basic household chores to a good-but-not-great level.

I don’t live in the United States, so I don’t hold out hope for seeing one in action any time soon. I’m also not exactly sure when teleoperation will become full automation. Maybe a few years. Maybe sooner. But even if I could get my hands on a completely autonomous bot, I wonder what it would be like to have an electronic footman that lives in my house and does all the stuff I don’t want to do. Sure, a humanoid robot would (probably) save me more time than not, but I suspect it might be a strange experience to boss around a thing cosplaying as a human every day.

If our character is shaped by our habits, then it seems to me that interacting with a bipedal robot on a daily basis would be pretty relevant for the type of person I am and would like to be in the future. Should I start mistreating my new guest — or if I get used to commanding a human-like thing that always obeys — then I might find that style of interaction rubs off on my personality. I’m not saying it will make me evil, but on some level practising this kind of domination strikes me as Not Good for the soul.

Of course, it might be fine. Maybe I’d get used to it quickly, and it wouldn’t have too much of an impact on the sort of person I am learning to become. In either case, I won’t know what it means for me or anyone else until we give it a go.

Subscribe now

Objectification

Is it wrong to treat an inanimate object badly? In some respects no. A rock doesn’t have a sense of interiority, so you don’t need to worry about hurting its feelings. If there is no subject of experience on the receiving end, then there is no moral patient to fret over. There’s nothing to wrong, nothing to injure, and no duty owed. At that level, treating an inanimate object badly is simply not a moral act. It is value-neutral, like clearing a fallen branch from a path or dismantling a broken chair.

This is all well and good, but it doesn’t tell us much about the person treating an object badly. From this perspective, we might still worry about what kind of person we become by taking pleasure in destruction. Throwing a rock into an empty patch of dirt may not exactly be a morally troublesome act, but what about if you threw it somewhere more interesting, say in the direction of a gravestone?

Even if it does little damage, almost everyone recognises that this would still be a kind of desecration. Clearly the stone still feels nothing, but the act signals contempt toward the human world the stone belongs to. It violates a norm of care that flows outward from the living and the dead alike.

These kinds of acts reveal something about who we already are, but they also shape our growth by accustoming us to certain ways of being. When we rehearse indifference toward something that carries significance, we become the kind of person for whom indifference comes naturally. In that sense there’s a harm taking place to the self who becomes habituated to treating the world as something beneath them.

Then we have more sophisticated artefacts like, for example, a common household vacuum cleaner. You probably wouldn’t destroy your own, partly because you might need to hoover something up later but also because doing so would feel petty and self-corroding. Just like flinging a stone in a cemetery, the act changes who we are for the worse. Maybe not by much, but enough to matter with enough repetition.

But humanoid robots aren’t vacuum cleaners. These are things that will live alongside us, proxies for real people that we interact with as if they were social partners. This relationship strikes me as different in kind to the vast majority of tools and technologies that we have at our disposal.

Even if you know your new companion is a machine, it’s still a person-shaped thing that elicits scripts of command, deference, status, greeting, blame, and praise. That means your mind treats it as a social partner by default, but it also means that every interaction is coloured by a posture of mastery. You issue orders without negotiation, expect compliance without comment, and correct behaviour without apology. All the while, you are practicing being a person who takes compliance to be the natural order of things.

To be clear, my concern here isn’t that the robot feels but that we respond as if it belongs in the moral space normally occupied by persons. I am not thinking about people forming sentimental attachments that lead to over-reliance (that is one scenario, but not the most interesting one). The deeper issue is the kind of stance we learn to inhabit.

So, we might distinguish two different forms of relation:

• The instrumental, wherein the human form is used to secure trust and ease of interaction. The robot is still treated as a tool, but one that works better because it feels familiar. This provides a kind of psychological leverage in that the design nudges us into a cooperative stance.

• The moral, wherein we begin to treat it as a quasi-subject that sits inside the space we normally reserve for persons. Once it occupies that zone, our behaviour towards it becomes expressive. In engaging with it, we are practising a way of relating that, over time, changes who we are.

The latter dimension has animated discussions about our relationship with technology for the better part of two thousand years. Its roots go at least as far back as Plato, who described technology as a form of craft knowledge that shaped both product and practitioner. The cobbler’s téchnē produced shoes, but it also cultivated habits of judgment about fit, durability, and beauty; the navigator’s téchnē guided ships, but it also demanded an attunement to winds, stars, and currents.

In this framing, technology is something like a “training ground” or a set of practices that form the character of those who wield it. Technology externalises human capacity, but it also bends those faculties back towards us by fostering new dispositions and habits.

Aristotle argues that character is moulded by habit, that over time your actions in the world form the essence of who you are. For the man the early moderns called simply The Philosopher, the self is built one act at a time. He famously reminds us that we become just by doing just acts, wise by doing wise acts, and brave by doing brave acts.

But what about when people look like tools and tools look like people?

Humans are, after all, predisposed to treat anything with eyes and a voice as a social partner. We respond to appearance as if it indicates personhood, we extend the grammar of interaction to anything we can, and we adopt the stance that normally accompanies a face-to-face encounter if the situation allows for it. Some people soften their tone and say “thank you” to a voice assistant on their phone, even though they know perfectly well there’s no one on the other end. Others do the same with ChatGPT, though there is some logic here insofar as politeness often produces better responses.

The point is that human-like cues pull us into patterns of social behaviour. Given these robots are about as human-looking a form as we can imagine, we should expect them to stimulate our social reflexes and modify our expectations accordingly. Provided enough time, expectations become habits and habits become character.

There is a second concern here, adjacent to character but distinct from it. This issue deals with the nature of shared life that requires us to encounter other wills and adjust to them. Freedom is in one sense the skill of navigating a world full of other agents, each with claims and desires of their own.

If we spend enough time commanding a thing that always does what we ask, we may come to see effortful negotiation as an irritation and other minds as obstacles. A life without friction may feel pleasant, but it also risks dampening our sense of what freedom really is: the discipline of sharing a world with other beings.

Benevolent authority

When I’m writing about AI and philosophy I often find myself circling something that one could call the “skill issue” objection. This basically holds that people are pretty good at figuring stuff out for themselves and concerns about waning autonomy in the era of AI are overplayed. It’s not that deep, buddy.

In some ways, I have a soft spot for this idea. It’s true that most people can separate play-acting from real life and that we don’t instantly absorb every influence in our environment like sponges. This comes down to the nature of the self, which needs to be both flexible enough to accommodate change when experiencing new things and stable enough to avoid an about-face at the drop of a hat.

We’ve been here before insofar as servant societies of the past also supported civic virtue. Comments on the obvious shortcomings of these particular social relations aside, the butler didn’t corrupt the statesman and the aristocrat had a thing for civic society. This tells us that hierarchy and assistance do not automatically corrupt character, that you can maintain a semblance of virtuousness so long as authority is exercised with restraint and dignity.

Nor is it obvious that delegation is always bad news for becoming good people. Much of human achievement rests on being relieved of drudgery so we can spend time on the good stuff of judgement, creativity, and civic engagement. In a world already full of service relationships (e.g. apps, assistants, and actual people who help us) most of us somehow manage not to become petty tyrants.

The question with humanoid robots is not “will they deform the self by default?” but rather “how do we govern them in a way that makes us better?” In the best case, owning a humanoid robot and treating it well could actually allow us to grow by cultivating a kind of benevolent authority.

Aristotelian ethics describes this dynamic as oikonomia or “proper rule”. It suggests that some types of virtue are expressed through right use of power, that the point is not to renounce authority but to wield it in a way that disciplines the self as much as it directs others. Augustine argues something similar by insisting that power is only just when guided by “rightly ordered love”. If we are to rule over others, we must rule the self first.

Humanoid robots are going to eventually stand in for people. Maybe not right now, but likely one day in the not too distant future. When that moment arrives, many us will have a thing that walks and talks like a human that we can command to do our bidding. If we treat them with respect, we will become better for it; if we treat them with contempt, we will be the ones who suffer.

AI developers have a style. Badge logos and blocky text are a given, but even promotional materials have converged on the same type of vibe. The typical advert uses some combination of a smiling person talking to a phone, knowing replies by the assistant delivered in a friendly patter, and the dum-tsh-dum of a snare drum as the camera skips from bike-shed to bistro.

Anthropic’s recent effort is better than most, jettisoning the happy-go-lucky aesthetic for a mish-mash of falling pianos, country road-trips, neolithic cave paintings, deep sea exploration, and stellar phenomena. The idea is that AI is useful in the places you can imagine and in many others you can’t, that despite the doom and gloom so pervasive in Western societies “there has never been a better time to have a problem”.

That’s certainly true in some respects. Today’s models are more or less the stuff of pulp sci-fi dreams, friendly golems that have something to say about whatever question you pose (so long as you are careful to abide by the usage policy). Notwithstanding the protests of those who think the entire AI project is made of smoke and mirrors, millions of people seem to agree that they have problems that thinking machines can solve.

But in another sense, there has never been a worse time to have a problem. After all, the point of problems is to work them out ourselves rather than have someone solve them for us. As the old saying goes: “If you give a man a fish, he may eat for a day. If you teach a man to fish, he eats for a lifetime”.

Just doing stuff

The aphorism of the moment holds that “you can just do stuff”. A favourite of Silicon Valley, the phrase embodies a strain of thinking that distinguishes between “high agency” people and those who don’t have the right stuff. On one level this is a truism. You can of course do many things, some of which are meaningful and some of which are not. It’s a great time to have a problem, you just need to roll up your sleeves and get stuck in.

Agency here is about freedom, the leeway to do what you want and act on your desires. Freedom, though, has a funny habit of decaying into licence, the doing of whatever comes first to hand without reflection on its worth. If you spend your time “just doing” pointless things, that doesn’t strike me as particularly agentic.

What makes the mantra seductive is that it spares us the labour of asking which things are worth doing. In a culture obsessed with output, action becomes its own reward. Better to do something than nothing, to be in the arena than sitting in the crowd.

The cult of agency is a counterweight to a world of bureaucracies and stasis, a place where “nothing ever happens”. The ability to act at all can feel like a triumph, and so to take action is to stick one in the eye of institutional inertia. The language of agency thrives in certain corners of the internet because it assures us that we have what it takes to strike out somewhere, anywhere, on our own.

A more generous reading of the “just do stuff” meme is that implicit in its logic is that choice alone is not enough, that you do have to pick and choose wisely. A supermarket aisle may present a hundred brands of the same product without making us wiser about what we want or why we want it. The high-agency move would be to pick the perfect item, or better yet start your own grocery chain from first principles.

We might say that action should lead somewhere worth going, that movement is a means not an end. If this is true, then we know why the ideal of agency feels incomplete: it describes the ability to act but not the standard by which action is judged. Without that end, its easy to mistake momentum for direction, novelty for growth, and busyness for a life well-lived.

So agency needs direction, but how do we know where to focus our efforts? You figure it out by knowing the kind of person you are today and the kind of person you want to become tomorrow. This is better, but now we’re no longer talking about agency in a strict sense. We’re in the land of autonomy, the cultivated capacity to live well by reflecting on the type of person we want to be.

Autonomy is about deciding which things are worth doing and then binding yourself to that decision when appetite, novelty, or fatigue threaten to take you somewhere else. It’s about not-doing as much as it is doing. To live with autonomy is to set the rules by which competing desires are brought to order so a person can act for the better.

You practice autonomy by noticing your impulses and testing them against a standard you chose. One way to imagine the split is to think about first and second order preferences, where the former concerns what you want right now and the latter describes the kind of person you want to be. If you “just do stuff” in service of your proximate wants, don’t be surprised when you feel something is still missing even after you founded that company or wrote that book.

The shape of problems

Figuring stuff out for yourself has a practical element (in that it is the condition of knowledge) and a moral element (in that it trains you to become the kind of person you want to be). Plato’s Apology famously gives us Socrates’ claim that “the unexamined life is not worth living”. In this framing, virtue is a product of questioning because it forces us to test our assumptions and to reform our character.

The mathematician George Pólya said solving a problem using “your own means” trains the habits of reason and allows the doer to become more than they were. What he meant was that the value of problem-solving lies in the the struggle, that each attempt at reasoning leaves behind the residue of skill. It gives you a sharper sense of what counts as a good and a clearer picture of what kind of thinker you are, so that the next problem — and the one after that — gets easier.

When we ask an LLM to solve our problems, we get a serviceable answer at the cost of truly understanding how we got there. Knowing, in other words, is not the same as growing. Every time we use ChatGPT to work something out for us we deprive ourselves of the opportunity to become a little bit wiser. People are already outsourcing cognitive labour to large language models with little regard for debates about whether AI can “think” or not.

The rub is that AI doesn’t only help us do things. Clearly in some instances it does, like teaching us a new skill or surfacing sources of information that we might not have seen. But it also proposes what to do, how to do it, and why we should care. This shift moves us from assistance (a tool serving chosen ends) toward deference (something that proposes ends we adopt without thinking).

Models choose what you see first, how options are ordered, which interpretations are offered as “reasonable”, and which are not even offered for consideration in the first place. A recommended route, a suggested reply, or a pre-filled summary frame the terms of engagement by providing the architecture under which we make choices. They don’t always pick what you eat, but they forever set the menu.

Systems infer objectives from us and optimise toward them. Often that takes the form of maximising engagement, even though large language models are not explicitly designed with this goal in mind. Their stickiness in part flows from the post-training procedures designed to turn the base model into a chat assistant. It’s pretty easy for “helpful, honest, and harmless” to become “the kind of thing I quite like talking to all day”.

You might say that this problem is something that all technologies face, that we’ve been here before and the worries were overblown. The pen doesn’t tell us what to write anymore than the calculator tells us what to add or subtract, right? The difference is that while all technologies in some sense structure our actions — the wheel made certain journeys possible and cartography influenced patterns of trade — we don’t outsource the habit of thinking to these artefacts.

It’s also the case that some off-loading is beneficial. Humans have limited cognitive bandwidth, and spending it on memorising every route or re-deriving calculus is probably not the best use of that mental currency. The trick is to distinguish between delegation that clears space for higher forms of judgment and delegation that spells trouble for the work of judgment in the first place.

The classic justified truth belief (JTB) theory of knowledge describes its subject as a mental representation that corresponds to reality, one that is underwritten by a justification. It’s essentially a mental mirror of the world that is true and warranted. Assuming that JTB tells us something useful about how knowledge is made, then the problem that AI poses is clear enough.

AI can deliver a proposition that happens to be true, but if you have not traced the steps, weighed the reasons, and ruled out the alternatives yourself, then that knowledge isn’t really yours. I’m not so worried about machines making mistakes, but I do wonder whether the act of deference erodes the habits that let us truly say we know.

We might even say that autonomy reveals itself most clearly when tested against the temptation of deference. AI endangers self-rule but it also provides the conditions under which it can be tested, offering each of us a chance to practise rejecting the easy answer and favouring the harder work of thinking.

I use ChatGPT or Claude most days, and I’m probably as guilty as anyone for asking the robot about things I could probably have figured out for myself. I don’t try to police my use, but I do try to think deliberately about it. One difference lies in letting it clear space and letting it fill space for me. The temptation is always toward the latter, because it’s easier to accept answers than to wrestle with problems.

But to live well with machines is to insist that they serve our efforts at growth rather than replace them, that they enlarge the field for judgment instead of shrinking it. The task of becoming the person you want to be — the kind who can judge, discern, and act — cannot be outsourced. It has to be practiced by each of us, with all the false starts and frustrations that practice entails.

Subscribe now

“You see that white part of the sky, called the milky-way. Can you guess what it is? An infinity of little stars, invisible to our eyes on account of their smallness, and placed so close to each other that they seem but a stream of light. I wish I had a telescope here to shew you this cluster of worlds.”

Bernard Le Bouyer de Fontenelle, Conversations on the Plurality of Worlds (1686)

Bernard Le Bouyer de Fontenelle was a French author and philosopher. He wrote history and theatre, and even tried his hand as a lawyer before quickly deciding the legal profession wasn’t for him. Fontenelle was the model of an Enlightenment man. He believed in the renewal of human will and reason, and argued with gusto against academic colleagues who thought the great works of the past could never be equaled.

In 1686, the Frenchman published Conversations on the Plurality of Worlds. Written as a string of exchanges between a philosopher (a thinly veiled stand-in for Fontenelle) and an intelligent woman called the Marchioness, the work established the template for Enlightenment extraterrestrial discourse by recasting astronomy as a question of humanity’s place in the cosmos.

The first evening starts with the Marchioness and the philosopher on an evening stroll. As they watch the Moon and stars, Fontenelle’s philosopher bashfully admits that “I have taken it in my head that every star may be a world”. His trepidation flows from the recognition that radical ideas often run against the grain of human nature, that people like to cling to that which flatters their pride.

Our philosopher explains that astronomers held on to the old Ptolemaic model of the heavens because they wanted to put themselves at the centre of the universe, much like the courtier who tries to place himself in the most prominent position at court. Copernicus won out, he tells us, because the Ptolemaic system buckled under the weight of its own complexity. When Mars appeared to move backwards in the sky, astronomers explained it by saying the planet circled on a little ring, which in turn circled on a bigger ring, and so on. These epicycles multiplied until the model eventually looked like a funhouse mirror version of Ptolemy’s original scheme. The reason we were hesitant to accept it is for fear of what a heliocentric account means for our place in the universe.

The Marchioness isn’t moved by his argument. She asks: “Do you suppose I feel humbler for knowing that the earth goes round the sun? I assure you I esteem myself just as highly as I did before.” This is the essential question of the book, one born of the Enlightenment confidence in reason and nature’s order. We only believed Copernicus, he says, because the system of nature compelled us to. Yet in doing so we learned to accept something that cuts against the human instinct to put ourselves at the centre of the cosmos.

Conversations on the Plurality of Worlds is a negotiation between science and the human condition. If the stars are other worlds, then the universe is richer than we ever imagined. That which seemed threatening — humanity’s downward movement in the celestial hierarchy — seemed to Fontenelle to demonstrate that reason could accommodate dislocation. He saw this upheaval as proof that reason could bear uncomfortable truths, that humans could (and should) draw dignity from their new place in the pecking order.

Subscribe now

Is there anyone out there?

Today, speculation about aliens is old hat. We see it in popular media, in research efforts like Breakthrough Listen, and in academia where studies of extrasolar beings are finally approaching something like scholarly respectability. Whether you have a strong opinion on the matter or not, all of us are well aware of the possibility of life out there.

It’s easy to forget that wasn’t always the case. We take discussion about little green men for granted, but our ancestors probably wouldn’t understand the question. Then again, the preoccupation with ET certainly feels like a very modern phenomenon. It’s the stuff of Hollywood, pulp sci-fi magazines, and turn of the century novelists, isn’t it?

Aliens surely loom larger in our collective cultural imagination than ever before, but serious consideration of extraterrestrial life as we might understand it has been hundreds of years in the making. The roots of our moment begin with the convulsions of early modern science, when the telescope made the stars look like other suns and the planets look like other earths.

For most of human history the heavens were the realm of gods, spirits, and influences; only in the seventeenth century did people begin to ask, in a recognisably modern way, whether those distant worlds might harbour other kinds of life. Bernard de Fontenelle was one of the first to popularise that shift. In Conversations on the Plurality of Worlds, he wondered whether the planets of the solar system contained life, and even speculated that every star in the night sky contained a solar system like our own.

But he also knew that alien life may not look like us. On possible lunar inhabitants, he wrote: “I say there are inhabitants, and I likewise say they may not at all resemble us,” and that any alien life must adapt to its own planetary conditions: “when I affirm that the moon is not peopled by men; you will see that according to the idea I entertain of the endless diversity of the works of nature, it is impossible such beings as we, should be placed there.”

His modesty masked confident assertions about extraterrestrial existence based on the principle of plenitude, the belief that nature abhors waste and fills all possible spaces with life. What made Fontenelle’s case so effective was both the boldness of his claim and the elegance of its presentation. He wrapped unsettling ideas in polite conversation, choosing as his interlocutor a witty, curious woman. In doing so, he signalled that reason was not the preserve of scholars alone, but something that anyone with curiosity could exercise. As he explained:

“In these Conversations I have represented a woman receiving information on things with which she was entirely unacquainted. I thought this fiction would enable me to give the subject more ornament, and would encourage the female sex in the pursuit of knowledge, by the example of a woman who though ignorant of the sciences, is capable of understanding all she is told, and arranging in her ideas the worlds and vortices. Why should any woman allow the superiority of this imaginary Marchioness, who only believes what she could not avoid understanding?”

Plurality of words

Fontenelle wasn’t the only Enlightenment thinker wondering about alien life. The ‘plurality of worlds’ debate became a live issue as scientists continued to spy new solar objects down the end of their telescopes. Naturally, Enlightenment thinkers moved to grapple with the questions that flowed from these observations. Were humans unique in the cosmos? How would extraterrestrial life affect Christian salvation doctrine? What moral obligations might exist toward rational beings on distant worlds?

The Dutch polymath Christiaan Huygens' posthumously published Cosmotheoros in 1698, which provided a systematic scientific treatment of extraterrestrial life by applying Newtonian physics and observational astronomy. Unlike Fontenelle's accessible dialogues, Huygens wrote a dense tome that established methodological principles that would influence subsequent astronomical speculation.

His fundamental argument rested on the Copernican principle: “A Man that is of Copernicus's Opinion, that this Earth of ours is a Planet, carry'd round and enlighten'd by the Sun, like the rest of the Planets, cannot but sometimes think that it's not improbable that the rest of the Planets have their Dress and Furniture, and perhaps their Inhabitants too as well as this Earth of ours.” This style of inquiry, one that connected reason and analogy, became the dominant approach in Enlightenment extraterrestrial discourse.

Huygens provided remarkably detailed speculation about ‘Planetarians’ based on functional reasoning about intelligence and technology. He argued they must possess manipulative organs because “without their help and assistance men could never arrive to the improvement of their Minds in natural Knowledge.” Perhaps his most famous idea was that the inhabitants of Jupiter must cultivate something like hemp for rope-making in their sailing ships, an assumption that demonstrated the period's confidence in analogical reasoning and its assumption that technological development followed universal patterns.

Others discussed extraterrestrial visitors as a form of social criticism. Voltaire's Micromégas from 1752 features a giant from Sirius (Micromégas, 120,000 feet tall) who visits Earth with a Saturnian companion (a puny 6,000 feet tall) in a text that sought to provide a cosmic perspective on human vanity. When Earth's inhabitants claim the universe was created for their benefit, “the two travelers fell on each other, choking with laughter”.

Of course, no treatment of life amongst the stars would be complete without religion. American founding father Thomas Paine deployed extraterrestrial life as a central argument against traditional Christianity in The Age of Reason, published in three volumes between 1794 and 1807. Paine's core argument targeted Christianity's cosmic exclusivity:

“Though it is not a direct article of the Christian system, that this world that we inhabit is the whole of the habitable creation, yet it is so worked up therewith, from what is called the Mosaic account of the Creation, the story of Eve and the apple, and the counterpart of that story, the death of the Son of God, that to believe otherwise, that is, to believe that God created a plurality of worlds, at least as numerous as what we call stars, renders the Christian system of faith at once little and ridiculous, and scatters it in the mind like feathers in the air.”

Paine was not out to abolish belief in God, but he was out to reform it. If revelation on Earth was the only path to salvation, what of the innumerable other worlds? To posit a separate incarnation for each, he argued, was absurd; to limit salvation to Earth was parochial. Faith in the creator must reflect the immensity of creation, and Christian doctrine ought to accommodate the true scale of the universe.

Finally, Immanuel Kant integrated extraterrestrial speculation into his comprehensive cosmological system. His Universal Natural History and Theory of the Heavens from 1755 examined solar system formation, arguing that the same processes that produced life here would operate elsewhere throughout the universe.

In Critique of Pure Reason, he wrote: “if it were possible to settle by any sort of experience whether there are inhabitants of at least some of the planets that we see, I might well bet everything that I have on it. Hence I say that it is not merely an opinion but a strong belief (on the correctness of which I would wager many advantages in life) that there are also inhabitants of other worlds.”

The idea constituted a hierarchical arrangement of planetary inhabitants based on distance from the Sun. Beings on planets closer to the Sun would be of a denser and more refined nature, while those on distant planets would be made of lighter stuff. All would possess reason, but their physical forms and capabilities would vary according to planetary environments.

Kant used the possibility of extraterrestrials less to describe aliens themselves than to clarify what it meant to be human. For Fontenelle it was a way to charm readers into accepting displacement, for Huygens to prove the universality of nature’s laws, for Voltaire to puncture vanity, and for Paine to expose the limits of revelation. In each case, other worlds served as proxies for disputes over knowledge, power, and salvation.

New horizons

In Conversations on the Plurality of Worlds, the Marchioness tells the philosopher “You are making the universe so unbounded that I feel lost in it; I don't know where I am”. The proper response, he insists, is to feel the opposite:

“For my part, said I, I think it very pleasing. Were the sky only a blue arch to which the stars were fixed, the universe would seem narrow and confined; there would not be room to breathe: now that we attribute an infinitely greater extent and depth to this blue firmament, by dividing it into thousands of vortices, I seem to be more at liberty; to live in a freer air”.

Extraterrestrials were a rhetorical instrument. They allowed Enlightenment writers to weaken the idea of divine privilege and to argue for the universality of reason, law, and moral order. Speculation about other worlds was a way of imagining a universe without exemptions, a politics without ecclesiastical hierarchies, and a humanity defined by its participation in a community of rational beings.

In embracing the plurality of worlds, Enlightenment thinkers completed the Copernican revolution in the cultural imagination. Its ‘principle of mediocrity’ — the claim that Earth is not special, that what happens here is likely to happen elsewhere — was the scientific manifestation of the Enlightenment’s organising principle. Once you accept that the same laws of nature apply throughout the cosmos, you undercut the idea that anyone ought to benefit from a pre-ordained position on Earth.

Subscribe now

In Either/Or, Søren Kierkegaard tells us about life’s great enemy. It’s not pain or suffering, sin or despair. It’s not even failure or death. As it turns out, our true nemesis is boredom. He calls it ‘the root of all evil’ because it describes a mode of living where the habits that once formed character no longer continue to do so. Past actions stop making those same actions in the present feel significant, shaking off the sense of progress or growth that previously defined them.

Soon enough, you don’t recognise yourself in your own decisions. You might counter that maybe that’s okay, as you can always find new routines and habits. But not so fast. Kierkegaard also thinks boredom applies to the new as well as the old. Fresh experiences are meant to unsettle, to introduce friction, and to make us reconsider what we thought we knew. Under boredom, novelty is stripped of that power. It becomes a distraction, something briefly stimulating but quickly assimilated without any change in how you see yourself or the world.

The aesthetic life craves stimulation, but diversion has an annoying tendency to harden into the repetitive or mundane; the ethical life depends on habit, but habit without true renewal likes to decay into tedium. His salve is a ‘rotation method’ where we learn the capacity to let the old appear new and the new acquire depth.

In practice, that means learning to vary our perspective rather than our circumstances, to approach the same experience from new angles, and to linger with new experiences long enough for them to take root. It can be as simple as rereading a favourite book and noticing what strikes you differently, or taking a familiar walk with a new locus of attention. It can mean resisting the impulse to scroll for something ‘new’ and giving time for the novel thing you just discovered to mature into a deeper form of understanding.

What is at stake here is something like the freedom to truly know who you are and how to live. After all, freedom is not only the power to choose but the power to recognise yourself in your choices. By this reading, we might say a person is only practising true autonomy — the cultivated capacity to deliberate well about how to live — if their judgments are the sort they would continue to endorse after putting them to the question.

Philosophers call this ‘erotetic equilibrium’, the idea that a judgment counts as autonomous only if it can withstand the twin forces of reflection and deliberation. As Kierkegaard sees it, the threat of boredom is that it compromises this settlement as the familiar no longer feels grounded and the novel no longer feels renewing. Put in other terms: autonomy requires a rhythm between familiarity (in the form of stable, habituated judgment) and novelty (through disruptive experiences that reopen old encounters).

Today, our lives are governed by technology. We spend our time listening to music or watching videos served to us by algorithms, with the average person logging roughly seven hours looking at screens of various sizes. Not all of our time spent on a phone or a laptop is shaped by AI, but even just accounting for social media that takes us to something like two hours of every day at the mercy of recommender systems that govern what we experience.

Algorithmic recommendations are a boon for Kierkegaardian boredom. They interrupt the rhythm between old and new by systematically skewing novelty in favour the already-known and familiarity towards the already-consumed. In doing so, they erode the cycle of disruption and renewal that autonomy requires, leaving us with choices that are neither truly tested nor truly sustained.

Subscribe now

Boredom machines

Most recommender systems optimise for adjacency. Spotify’s ‘Discover Weekly,’ YouTube’s ‘Up Next,’ TikTok’s ‘For You’ feeds are all built to keep you coming back for more. The next item in the carousel is chosen because it is maximally like something you already enjoyed.

What looks like freshness is too often a very narrow kind of variation. These systems work by mapping your past behaviour into a dense space of similarities, then drawing from the tight cluster around it. The result is that the ‘new’ is different enough to feel like discovery, but close enough never to rock the boat.

That’s fine in some instances. Nobody needs their dinner playlist to feel like a challenge, and most people welcome a gentle learning curve when picking up a new app or tool. But when this becomes a dominant pattern of cultural life, the cost is that novelty never truly unsettles us and familiarity never deepens into mastery. We stay entertained, yes, but we do not grow.

Humans need what we might call the shock that opens the question. For Kant, this takes the form of the sublime, when the mind confronts something that does not fit its existing categories. The sublime is unsettling because it resists assimilation, yet in straining to make sense of it we discover that our capacity for reason and judgment exceeds what can be taken in by the senses. Without these disturbances, our judgments may never find themselves truly tested against what lies on the other side of what we know.

Heidegger describes the Unheimlichkeit or the uncanny moment when the familiar suddenly feels strange. Such moments matter because they interrupt the everyday flow and disclose possibilities we had previously ignored. Human potential requires this estrangement because we only know that our routines and preferences are genuinely ours when they face this sort of interruption.

We might comfort ourselves by urging that while algorithmic novelty isn’t always new, the systems at least allow us to become familiar with the things we already love. But that type of familiarity rarely matures into depth. Instagram recirculates the same handful of recipes, fitness routines, or travel spots, but the repetition doesn’t necessarily make you a better cook, athlete, or traveller. Familiarity here is broad but shallow, a surface-level exposure rather than patient discipline that ripens into something more meaningful.

Aristotle argues that virtue flows from habituation, the repeated practice of good actions until they become second nature. True familiarity approaches stability and depth only when repetition is combined with the good work of attention and discipline. The kind of algorithmic repetition we live with today looks like habit because it gives us the same patterns over and over, but it lacks its substance because the residue of experience is rarely incorporated into our character.

That’s because the logic of retention rewards what is easiest to consume again, not what is hardest to master. The system is designed to serve us repetition calibrated to hold attention rather than to cultivate depth. Where Aristotle thought habit was the slow conversion of action into character, platforms like to hold us in place rather than carry us forward as persons.

One way to think about this idea is the relationship between first and second order preferences, the difference between ‘man, I would love a cigarette’ and ‘I wish I could stop smoking’. First order desires are immediate and situational but second order desires are reflective, the stance you take on your own wants. In this framing, autonomy is not just acting on a first order preference but being able to align oneself with the second order judgments you endorse about the life you want to live.

Habituation is a bridge between these levels, with repeated actions gradually harmonising impulses and sustained reflection building character. The problem with algorithmic culture is that it breaks this connection. It gratifies first order preferences without giving them the friction that might force second order reflection. Clearly, that doesn’t happen all the time. Even within algorithmic culture one can pull away and use the same tools to pursue depth, like the person who studies guitar through YouTube tutorials or who joins an online community and learns to cook.

But these are acts of resistance, not dominant kinds of engagement.

Algorithmic culture dampens novelty and familiarity on their own terms, but what matters most is the negotiation of these two forces. It is in the back-and-forth between disruption and stability that our choices become truly our own. When novelty tests our habits and familiarity steadies our responses, we gain the chance to endorse our lives rather than simply living through them.

John Dewey, writing on education, put forward the idea that growth depends on the dual conditions of continuity and interaction. We develop when new experiences disrupt us, but only if they can also be tied back into what we already know. In this view, knowing oneself is about learning to weave these ideals into a life that we can call our own.

We need encounters that unsettle and habits that hold, moments that throw our judgments into question and practices that let them take root. To lose that balance is to risk the boredom Kierkegaard feared: a life rich in stimulation yet poor in meaning.

A different beast

Not all AI is made equal. We might interact with recommender systems on a daily basis, but many of us aren’t even aware that we’re doing so. Large language models feel different because they confront us more directly. They speak to us, take instructions, and generate responses to our feedback in the moment. These systems aren’t necessarily more fluid, but we are more aware of how these artefacts respond to human interaction.

We consult with them by treating the model as an oracle for information or advice, and we collaborate with them by enlisting them as a partner in drafting, editing, or brainstorming. But we also let LLMs play the parts we assign them — as tutor, friend, or opponent — and even hand off some tasks to them altogether. These modes of interaction promise control (we set the terms of the interaction) and companionship (we enter into a dialogue with the machine).

At first blush, they look like autonomy preserving custodians of algorithmic culture because we manage inputs and decide whether to accept or reject what is offered. Yet we know that the range of responses is bounded by training data, defaults, and hidden constraints that shape the choices we appear to make. I have lost count of the number of times models surface the same idea in different contexts, and we are all too aware of the linguistic tics that make LLM-generated text stand out.

Large models can in principle explore a giant space of possibilities, but in practice they tend to organise around the same set of patterns. At the point of use, these systems gratify first order preferences in a way that doesn’t guarantee the type of self-reflection we need to grow. A student may turn to a model in order to ‘learn,’ but the act of outsourcing the work can shortcut the struggle through which understanding emerges. The immediate preference is satisfied, but the deeper preference — to know, to master, or to grow — may not get a look-in.

More deliberate modes of use can help. We might say that a student who asks for a sequence of questions to work through, or a writer who uses the model to expose weaknesses in their own draft, is using the tool to test and refine their deeper commitments. In these cases the model becomes a means of holding ourselves to account, of forcing our immediate desires to answer to the kind of person we aspire to be.

At its best, AI returns our words in unfamiliar shapes and forces us to clarify what we mean. In those moments the familiar is unsettled and the novel anchored. At its worst, the same system may take us closer to ‘autocomplete for life’, gratifying first order desires while leaving our deeper commitments untouched. Instead of testing our judgments, they tell us what we want to hear.

Sam Altman acknowledged these two types of usage in a recent podcast: “There are some people who are clearly using ChatGPT not to think. And there are some people using it to think more than they ever have before. I am hopeful that we'll be able to build the tool in a way that encourages them to stretch their bandwidth a little more”.

The idea of rotation is useful here. Language models help us grow when they take something we already know and reframe it in a way that brings forth a new perspective. When you get a response, better to ask the LLM to defend, refine, or contradict itself rather than taking it as gospel. That might seem obvious to some, but last year’s ‘Claude Boys’ phenomenon reminds us that people don’t always like to do that kind of work.

These ideas encourage us to remember that autonomy depends on the rhythm between novelty and familiarity. We need habits that hold and shocks that unsettle, practices that shape character and encounters that open the question. So long as we use them judiciously, large language models are commensurate with that account of autonomy. They can either gratify our first order desires in ways that leave us unchanged, or they can be turned into partners that test our ideas, sharpen our commitments, and force us to see ourselves anew.

The difference lies partly in us, but also in the technology itself. Some systems make renewal harder, though not impossible, while others open the space more readily. To live well is still to weave the known and the unknown into a life we can endorse, and that remains our task, whatever tools we choose to do it.

Subscribe now

Henry James’ The Beast in the Jungle follows a well-heeled American drifter in London called John Marcher. Over the course of the novella, Marcher agitates about the coming of a ‘beast’ poised to emerge from the undergrowth and destroy him and all that he holds dear.

In the closing moments we find the beast is a fiction; or rather, the anticipation of the monster’s coming is beastly in that Marcher lets it consume his life. He ignores his love, his friends, and his career in preparation of the creature, only for those actions to provoke the monster into being.

James’ little book is in that sense a self fulfilling prophecy, a cautionary tale about foresight and control. The essence of the story is common enough to be inaugurated as the Eighth Basic Plot: Chaucer’s Pardoner’s Tale sees those hunting death become murderers; The Appointment in Samarra finds a man running from death only to meet him at his destination; and in Boccaccio’s Decameron a man’s jealousy over losing his lover causes her untimely demise.

These are all Oedipal tragedies of sorts, though in each case the doom is more literal than Marcher’s fate in The Beast in the Jungle. In the classic version of prophecy gone sour, our hero’s folly is a desire to take action. In James’ book, stasis is the malady. We are shown a man afraid to travel, who fears love and intimacy, and who can’t bring himself to live life in case something goes wrong.

Subscribe now

Future shock

The things we call ‘technologies’ are ways of imposing order on an unruly world. They are artefacts, devices, and systems that contain possibilities for structuring human activity. Deliberately or by chance, consciously or unconsciously, societies select certain frameworks that determine how people work, communicate, travel, and play.

These structures are technological, and they in turn shape our ability to live in the world and produce new kinds of structures. Recognising that fact is not the same as advocating for a kind of technological determinism, the idea that technological development is the basic currency of change and that humans have no choice but to sit back and let it happen. Determinism is simply too all-encompassing a theory of progress, one that does scant justice to the choices that arise as we design, build, deploy, and configure our technologies.

We know that technologies do not materialise whole, that they are assembled inside labs, garages, parliaments, and patent offices. A technology’s function is a standing preference manifested in the world. When an engineer decides to design a safety guard to prevent a saw from touching a workman’s hand, the preference is ‘saw stops before contact’ over ‘saw cuts hand’.

The parts of our woodcutting machine continue to enforce this preference long after the thing has been designed and put to use in lumber yards. It keeps happening, every time it is used, whether or not the person slicing logs knows it or not. We might say that preferences or ‘values’ live in technology, which is one reason that technological determinism falls apart under just a little bit of scrutiny. QWERTY was not the fastest keyboard but the one that had a layout that seemed to prevent typewriters from jamming. The choice stuck because salesmen, schools, and secretaries made it stick.

When engineers wrote the GSM mobile phone standard in the late 1980s, they famously added in a 160 character Short Message Service (SMS) as a low priority maintenance channel so network staff could ping each other with status alerts. It wasn’t marketed, priced, or even imagined as a consumer feature. A contractor sent ‘Merry Christmas’ from his desktop to a colleague’s handset, and not long after curious users began trading notes.

What began as a backstage diagnostic tool morphed, through unplanned tinkering and uptake, into what was once the world’s favourite chat medium. The point here is not only that technology can change, but that social logics are often the drivers of that change. Taken in the round, it reminds us that any time you use a technology you are entering into a social negotiation with all those who had a say in its making and usage.

But let’s not get carried away. Technological determinism may be inadequate, yet so too is the view that technical things do not matter at all. It is deeply misguided to assume that once we locate the social origins behind a particular technology, we will have explained everything of importance.

By parcelling the day into equal and audible hours, the mechanical clock let factory owners synchronise shifts. Nobody decreed that the bell must rule the worker, yet once the hours of the day could be precisely tracked the temptation to regiment wasn’t too far behind. Gears didn’t force obedience, but they create affordances and invite patterns of use that can be hard to resist. You can, in theory, ignore a seatbelt reminder that asks you to buckle up, but in practice most of us would rather give in.

Design choices at T₀ become path dependencies at T₁ and common sense at T₂. If we stop at the backstory — who funded the clock, who invented the seatbelt — we close our eyes to the way artefacts shape and are shaped by the movements of everyday life. Conversely, if we fetishise the thing, we overlook the social climate that birthed it and our capacity to reroute it if we choose to do so.

Directions of travel

In the Republic, Plato tells us about techne, or expert know-how of a craft like carpentry or surgery. For the Greek, techne brings with it a kind of automatic authority that flows from expertise. We submit to the surgeon on how to set a broken bone because she has the skills and knowledge to align the fragments and keep infection at bay.

Plato leans on that prestige in his masterwork, arguing by analogy that the city should likewise be steered by those with the requisite political techne. These are the philosopher kings who understand the true nature of things, so their expertise ought to ground the state’s right to rule.

To make his case, Plato asks us to think about a ship on the high seas. Large sailing vessels need to be steered with a firm hand, so sailors must yield to their captain's commands. We don’t expect ships to be run democratically because a vessel’s survival hinges on technically informed decisions, like how to trim sail in a squall or plot a safe course through shoals.

Plato goes on to suggest that governing a state is much the same. It is something rather like captaining a ship or practicing medicine in that it demands specialised knowledge and the wisdom to apply it judiciously. He returns to this idea in the Laws, where he compares his own work to that of a well established craft.

“The shipwright, you know, begins his work by laying down the keel of the vessel and indicating her outlines, and I feel myself to be doing the same thing in my attempt to present you with outlines of human lives.... I am really laying the keels of the vessels by due consideration of the question by what means or manner of life we shall make our voyage over the sea of time to the best purpose.”

Philosophers have a reputation for having their heads in the clouds, but there is some evidence that Plato did in fact seek to put his skills as a designer of political societies into practical effect. He famously travelled from Athens to the court of Dionysius the Elder, hoping to transform his host into a philosopher king willing to put the principles of political techne to work.

Plato treats techne as a model for political rule, honouring the shipwright’s expertise only insofar as it buttresses his claim that the city should be steered by those who know. In the Laws, he bars actual artisans from citizenship on the ground that their craft absorbs them wholly, and leaves no room for the higher labour of deliberating about justice.

In this model, hierarchy precedes technology. Authority is granted from above on the basis of wisdom, while the makers are banished from citizenship so they can focus on their craft and avoid upsetting the political applecart.

Of course, the opposite is also true. Technology constitutes political order just as surely as political order constitutes technology. In The Visible Hand, the historian Alfred Chandler argues that the expansion of the railroad in the 19th century shows how certain crafts grow their own hierarchies.

Railroads, he writes, could move freight across the continent in any weather, but speed was useless without an army of schedulers, track gangs, clerks, and district superintendents to choreograph arrivals, inspect boilers, and bill customers. Out of that practice emerged the first modern managerial pyramid, with rungs as rigid as any military.

The telegraph needs repeaters and time-zone standards; the power grid needs load balancers and dispatch centres; a cloud platform needs site reliability engineers, compliance teams, and a legal department. Each new layer of machinery widens the gap between operator and outcome, and that expansion calls forth coordinators to put the pieces back together.

Seen this way, techne is an engine that manufactures new politics in situ. The timetable does as much governing as the governor, and X dot com can settle arguments faster than a senate debate. Yes, we make technology — but technology makes us too.

Lost in the woods

There’s a meme about our current place in the ‘tech tree’, one that asks how it is that trillions of dollars of capital came to be expended in one of the largest programmes of investment in history. This story, the AI story, involves a combination of repurposed hardware, extremely rich companies, and mountains of data created by the growth of the internet.

Graphics processing units were originally designed for rendering virtual environments in video games. How fortunate that this architecture, created for performing thousands of operations in parallel, was exactly what deep learning systems needed in order to chew through data fast enough to make the magic happen.

Of course none of that matters if you have nothing to feed the networks. Garbage in garbage out may be true, but quantity seems to have a quality of its own. Still, to keep those loss functions down you need access to huge amounts of data. This is possible because the commercial internet, especially social media platforms, persuaded billions of people to publish text, images, and video as a side effect of trying to entertain friends or sell products.

Growing fat on targeted advertising, internet infrastructure, and consumer goods, those same firms piled up extraordinary cash reserves. When the time came, they could plough mountains of dollars into data centre construction, specialised chips, and research laboratories. A single US firm can now spend more on AI hardware than the UK does on defence.

Political orders embed themselves in the design and allocation of tools, while some tools push back by generating new political structures. We might say the techne of deep learning was socially selected, just as we might acknowledge that our place in the tech tree means the basic shape of frontier AI systems is unlikely to change much in the medium term. That isn’t to say that progress is sure to slow, but instead that we already know what AGI will look like if it’s built in the next five years (assuming the predictions of the US and UK governments are correct).

Whether by taking action or staying still, in the Oedipal tragedy fate always wins in the end. The weight of the future is simply too great to contend with, its power overwhelming for mere mortals. Determinists think something similar. They rightly point out that technology has a life of its own, but they are quick to forget that it is also enmeshed with the lives of others.

In James’ book, the future is paralysing. Our protagonist cannot see the wood from the trees, the ways his life is already changing as he frets over the coming beast. In our moment, we wonder whether the monster will arrive in 5 years or 50. Whatever happens — and whenever it happens — like John Marcher we’ll only recognise it with the benefit of hindsight.

Subscribe now

John Wilkins was a mover and shaker in the early years of the Royal Society. He was a clergyman and an experimenter whose passion project was ‘philosophical language’, a universal written system that could directly correspond with the structure of things in the world.

Wilkins wanted to turn the bones of language into an ontological framework for making sense of reality. His efforts remind me of noted postmodern linguist Plato, whose Cratylus put forward the idea that words must have intrinsic meanings. We are told that the Homeric hero Hector, for example, gets his name from the Greek verb ‘échein’ or ‘to hold’ because he was said to ‘hold’ the city of Troy as its great protector.

In his 1668 An Essay towards a Real Character, Wilkins introduced descriptive tables that showed how components of language could be used to classify certain animals. The word for 'elephant’ turns up as ‘zibi’, made up of ‘zi’ (the two letter root for every beast), followed by a ‘b’ (the consonant marking whole footed mammals), before finishing with i (the vowel assigned to the corresponding species in that row).

Like so many neat ideas, Wilkins’ philosophical language dissolved on contact with reality. It was too clever and too clumsy. Once you try to sort things into boxes, you soon find that the world has an annoying habit of contorting to avoid easy classification.

Argentine Jorge Luis Borges found Wilkins’ work in the 1940s, then famously sent it up by describing a ‘Celestial Emporium’ whose animal classes include ‘those belonging to the Emperor,’ ‘frenzied ones,’ and ‘those included in this classification.’ Borges’ point was that taxonomies are as arbitrary as they are brittle, that they tend to break the moment they are faced with a creature, a culture, or contradiction that doesn’t fit the scheme.

Wilkins thought his work could be a cabinet of cabinets, a scala naturae for the age of microscopes and coffee house empiricism. He lived as science wrestled with its Scholastic inheritance, a drive to fix the natures of things by figuring what they were and how they were related to one another. Our man pushed that logic to its obvious conclusion. If the world is orderly, then a language that mirrors that order must also be orderly.

Subscribe now

One of these things is not like the others

On 29 March 1823 a package from Sir Thomas Brisbane, the Governor of New South Wales, arrived at Edinburgh College Museum. Inside were two platypus carcasses, their ‘rostrum half dissolved, and the pile loose,’ as the curator’s assistant William MacGillivray grumbled in his log.

One went to the display case, the other to the Scottish anatomist Robert Knox’s dissecting table, where its curious mix of qualities proved inconvenient for every classification schema of the day. Knox found fur but no nipples. A keratinous beak and a cloaca, but no feathers to match. And a venomous spur without a cold-blooded body temperature.

There were lots of ways to cut the taxonomical cake, but the knife of choice came when Carl Linnaeus laid down the rules around a century earlier. In the 1758 Systema Naturae he offered a key composed of classes, orders, genera, and species, each demarcated by a handful of traits. Hair and teats? Mammal. Feathers and a beak? Bird. Scales and cold blood? Reptile. The attraction was its promise of mutual exclusivity: once a creature could be placed within one class, every other stayed limits. For a while it worked a charm, letting European naturalists sort the spoils of empire into their preferred locations.

By the early nineteenth century, however, the anomalies were coming thick and fast. Marsupials that suckled young yet carried their offspring in a pouch; microscopic euglena that prowled for food like an animal but carried chloroplasts like a plant; and the duck-billed creature that arrived in Scotland to confound the biologists. Each exception forced addenda, sub-orders, and awkward footnotes until the Linnaean grid was overrun by a patchwork of special cases.

To some extent the Victorians were alive to these anxieties. In A System of Logic published two decades after the incident in Edinburgh, John Stuart Mill argued that some groupings track real causal similarities while others are categories of convenience. After On the Origin of Species in 1859, the taxonomical project shifted away from fixed essences and towards a genealogical map of shared descent. In the post Darwinian order, classification was explicitly about relations and overlaps instead of Platonic blueprints.

As Ferdinand de Saussure took much glee in pointing out, a sign has no natural bond to its referent. Tree is not tree-ness in sound-form; it’s the noise we agree on because it isn’t three, free, or shrub. Meaning is the friction produced by contrast among signs. Vocabulary triangulates between differences and the essence of the thing is only stable insofar as it exists as what Ludwig Wittgenstein called a ‘family resemblance’ between things with overlapping similarities.

Moral philosophy by checklist

Values are the structure we impose on the messiness of the moral universe. They are meant to “capture collective wisdom about what is important in human life, in various contexts and at various scales” and help us sort the better from the bitter.

For large language models as in technology more generally, we appeal to ‘values’ as a source of illumination to help us puzzle through the most difficult questions and choices. Alas, the concept of ‘values’ is better seen as a symptom of confusion. We retreat behind values when we can’t find the right words for talking precisely about the most basic aspects of the human condition.

As Langdon Winner put it in The Whale and the Reactor almost forty years ago:

In a seemingly endless array of books, articles, and scholarly meetings, the hollow discourse about "values" usurps much of the space formerly occupied by much richer, more expressive categories of moral and political language. The longer such talk continues, the more vacuous it becomes, the further removed from any solid ground.

We are minded to believe that there have always been ‘values’ just as surely as there has been a long history of spirited discussion about them. Except that isn’t really true. People have always had commitments, responsibilities, preferences, tastes, aspirations, convictions and cares. But only in the last century or so has anyone bundled these things together as ‘values’ as we might understand them today.

Used as a noun, the word ‘value’ is an old term that has throughout most of its history meant ‘the worth of something’. Commonly the worth of an object in material exchange or the status or worthiness of a person in the eyes of others. The word properly enters social and political thought in the writings of eighteenth and nineteenth century political economists, most consequently via Adam Smith, David Ricardo, and Karl Marx.

For them ‘value’ meant the worth of a thing in a commercial sense, which is why a theory of value first appears wearing the clothes of economics. Later in the nineteenth century Friedrich Nietzsche commandeered the term to signify the sum of principles, ideals, and desires that make up the basic motivational structure of a person or people.

Nietzsche wrote about the need for umwertung aller werte or the ‘revaluation of all values’, a kind of controlled demolition of Christian morality. He wanted to tear down the moral order, sift through the rubble for anything still moving, and then rebuild a more life-affirming house from the ground up.

Later, Ralph Barton Perry proposed a ‘general theory of value’ that tried to give a reasonable account of the full range of human interests. Value is in this setting any object of note, whether that interest is aesthetic, moral, economic, or religious. He grounded these concerns in the life of instinct or desire, then cast ethics as a social technology for reconciling the inevitable clashes among them.

Even towards the middle of the nineteenth century, talk of ‘value’ was generally taken to be about some attribute of a given object. One might use or keep safe a thing because it had a certain value. Economic or sentimental, value was still value. We still accept this meaning, as say the ‘value of’ intellectual property or spending time with one’s family.

Today, you are just as likely to hear ‘value’ used to describe wholly subjective phenomena. People, groups, cultures, and even whole countries (British Values™ or American Values™) apparently have values that influence how they show up in the world.

These kinds of values are basically general dispositions, a semi-conscious filter of taste or conduct that reside in us rather than in the world. We do not cherish charity because charity is good; charity is good because our internal value set fires a positive signal when we see some philanthropy that we approve of.

All such things are personal sentiments don’t you know, despite the fact they can also be stretched across the full width of the nation state. You have your values just as I have mine. One community exalts self-reliance, another solidarity, a third ritual purity.

Our world is a values shop (not to be confused with a value supermarket full of discount deals), where we fill up the trolley with the values commensurate with internally held sentiments. Prices are strictly personal — your courage may be on two for one, my justice a luxury import — so haggling is futile.

The problem with this state of affairs is that it prevents us from thinking critically about the moral world. In the ethics of technology, things are rarely named outright as good, prudent, or admirable and courses of actions are seldom defended as fair or necessary. The winning move is to mumble about ‘values,’ as though the label itself ought to carry the day.

Keep off the grass

Values are a moral taxonomy, a set of friendly labels that lets corporations, governments, or individuals signal virtue without wrestling with the particulars. A list of values feels tidy, mutually exclusive, and reassuringly universal. But we know better, don’t we? Courage can too easily become recklessness, loyalty can clash with justice, and patience can take the edge off excellence.

For AI, critics and boosters both retreat behind ‘value alignment’ programmes that assume moral life can be rendered as a checklist — fairness, privacy, autonomy, and so on — and that the machine’s task is simply to occupy as many boxes as possible. You don’t need to say much about which values are preferable, you just need to cram in as many as possible to your taxonomy of virtues. In fact, if you just make sure one of them is pluralism you can call it a day.

The most basic facets of the human condition are easily swallowed by the value alignment project. Don’t think too hard about it. Better to concede that moral life has no rough edges and that the work of judgement is secondary. Who cares to ask what courage demands or whom justice serves when you can list pleasant sounding labels and pat yourself on the back for a job well done.

Behind the lists are ideals of good and harm, duty and power, claim and consequence. Those words bite because they force us to take sides and give reasons. They make the trade-offs real by reminding us that value alignment cannot in fact be all things to all people. Better than that, it forces us to concede that moral philosophy is more than ticking boxes.

Subscribe now

Teach First is a UK charity that drops graduates into education jobs. It’s generally considered to be a good organisation, one that I know from experience has helped lots of smart people build a career in teaching. The group has historically selected candidates based on written assessments, but earlier this month it said it was accelerating a plan to switch towards face-to-face interviews.

The reason? University graduates are using AI in applications, which predominantly take the form of written assignments where anyone can put ChatGPT to work. Patrick Dempsey from Teach First said the charity had seen around a 30% increase in applications so far this year on the same period in 2024, a development he primarily attributes to large language models:

“The shift from written assessment to task-based assessment is something we feel the need to accelerate…there are instances where people are leaving the tail end of a ChatGPT message in an application answer, and of course they get rejected.”

Other accounts paint a similar picture, with graduate employment specialist Bright Network reporting that the number of people using AI for job applications has risen from 38% last year to 50% this year. And why wouldn’t they? Applications are tedious at the best of times, never mind when the odds of success are long.

If everyone can write well enough to pass a hiring round or two, it follows that employers will change tack to focus on assessments where LLMs aren’t much use. While that might seem straightforward enough, we ought to remember that in-person tests are not exactly a direct replacement.

Those running the hiring process are no doubt aware that teachers are encouraged to use AI in the classroom, so in-person interviews already proceed on the basis that successful candidates will use LLMs when they get the job.

A pivot to face-to-face tests might simply ignore that reality, or it might just reflect the lack of compelling alternatives. But whether or not the move aims to approximate written tests via in-person assessment doesn’t really matter. After all, the nature of face-to-face tests means they also probe for verbal and social competencies.

That’s because the people who succeed will be the ones who interview best, who get on well with those asking the questions in a way that makes them think they’ll be suited to the job. I think about this as the vibes premium: the increase in value placed on subjective traits — charisma, manner, confidence, aesthetic, speech, and presence — as it becomes tougher to use technical measures to distinguish between candidates.

The idea is related to but different from ‘interpersonal skills’ that are also likely to appreciate in value in the age of AI. However, our vibes premium is less ‘problem-solving’ or ‘leadership’ and more simply ‘making people want me around’. There are lots of ways to make that happen, but they generally deal with personality traits and behaviours even softer than these ‘soft skills’.

Subscribe now

Applying for everything

It’s rough out there for graduates. Clearly good jobs have always been difficult to land, but I suspect competition has been stiffening for quite some time as barriers to entry have collapsed. More people apply to graduate opportunities thanks to an expansion in higher education, online application portals, social media job openings, and international mobility. Now candidates can use AI tools that let anyone sound polished, which in turn raises the standards of applications and makes it harder for any one person to land a job.

On the other side of the line, a few friends have told me they no longer plan to hire because their respective leadership teams are convinced AI can do the tasks that they might want a junior person to take on (though it is worth saying the picture is more complicated than that).

Whatever the case, a recent survey reported that vacancies for graduate jobs, apprenticeships, internships and junior jobs with no degree requirement dropped by 32% since the launch of ChatGPT in November 2022. The same poll says that entry level jobs now account for 25% of the market in the UK, down from 28.9% in 2022.

We have two forces at play, both of which may be driven at least in part by the emergence of powerful AI systems:

• Graduate job availability is falling. This might be because managers are automating the work or doing it themselves with AI, though it is also possible there are other labour market effects at play that are responsible.

• Graduates are using AI in applications, so writing samples converge around the same level of quality. This compression may happen elsewhere, but it’s especially influential for entry level roles where differentiating factors are harder to come by.

In some ways the emergence of powerful AI systems is deeply humanistic. In entertainment, I expect good taste and human-led curation to become more valuable in a world overflowing with slop. I also imagine the same is true for the workplace, where technical skills become less relevant versus the qualities that only humans possess.

To be clear, my view is that AI raises the floor for technical skills but doesn’t necessarily eliminate the ceiling. A sharp candidate who knows the domain, has original ideas, and uses LLMs well can still outpace someone who just pastes in a prompt. The best performing candidates might even find themselves slowed down by the technology.

Even so, we’re interested in entry level jobs and graduate roles. These are the places where more people now get passable technical skills than ever before, which means employers need to look elsewhere to distinguish between candidates.

Regardless of how good the models get in the near future, I don’t see them fully displacing humans across the board. Part of that is about allowing employers to maintain a clear locus of responsibility at work, so if an AI makes a mistake firms can point to someone holding the bag.

But it’s also because there are some roles where we want a human in the loop, even if we don’t strictly need one. People go to a doctor for reassurance as well as care. They want empathy, explanation, and the sense that someone is emotionally attuned to their issue. The same goes for teachers, therapists, lawyers, and countless other jobs whose value partly flows from the human touch.

Alas, that is probably little comfort to graduates finding it hard to land a job. Even if today’s impact is overblown, young people will be those who experience the economic impact of LLMs first because it’s easier for the technology to help low skilled people become average rather than the excellent to become brilliant.

This sounds worrisome, but it does mean that they are likely to be the first to respond to the new demands of employers. Things that centre the human, things that let them play well with others, and things that mean their bosses want them around.

New signals

That leaves us in an interesting place, one where technical proficiencies like writing are no longer reliable indicators of skill or effort. Firms of all stripes are hungry for new kinds of signals that they can use to sort people, and so we get a hard pivot to face-to-face interviews.

This is a type of assessment less interested in output than in character, a trend that is likely to become more popular as firms look specifically for traits that only human beings can provide. It’s a manifestation of a kind of neo-humanism premised on the idea that the more technically proficient machine outputs become, the more we value the ineffable.

Neo-humanism returns hiring to something closer to the old apprenticeship model, where being able to work alongside someone mattered more than being highly credentialed. And at the other end of the spectrum, we should remember that elite institutions like Oxbridge never stopped interviewing for some courses.

There's something appealing about jobs going to people who can actually collaborate and communicate rather than those who simply know their way around a marking scheme. But a world where success depends on being likeable in a fifteen minute conversation is going to be a brutal one for people to navigate, where the premium on hard work may not be what it once was.

Maybe the future pans out differently, though I wouldn’t be so sure. As technical skills become table stakes, knowledge work will be increasingly defined by the stuff that makes us human. Put another way: given enough time, AI will make personality hires of us all.

Subscribe now

Peter Kyle is the UK Secretary of State for Science, Innovation and Technology. Responsible for asking the machinery of government to foster, develop, and react to important developments in these industries, Kyle is a senior politician whose voice carries weight in Westminster.

He also thinks that artificial general intelligence (AGI) is coming. And soon. On a recent podcast, Kyle said: “I think by the end of this parliament we're going to be knocking on artificial general intelligence.” For those not familiar with the timetable of the British political system, that puts the arrival of AGI in 2029.

It’s a strange statement that can be read in a few ways. Maybe Kyle doesn’t really believe what he’s saying or maybe he’s parroting lines that he’s heard from American politicians. I take Kyle to be intelligent and I don’t know what he has to gain by putting these remarks out there if he doesn’t believe them, so I’d be minded to give him the benefit of the doubt and grant that he believes his own forecast.

Another reading might stress that Kyle has a very specific view of what ‘AGI’ means that doesn’t really correspond to how people generally think about the technology. He followed up his statement with the suitably cryptic: “I think in certain areas, it will have been achieved,” which suggests he’s talking about systems with human-level performance in some circumstances but not in others. The problem with this interpretation is that it glosses over the ‘general' in artificial general intelligence (not to mention that today’s systems already exceed humans in some domains).

Option number three is that Kyle thinks of AGI broadly like most other people in the field — a system capable of conducting the majority of cognitive tasks that a human can — and does indeed think we should expect a system like this in just a few years.

For the purposes of this post, I am going to assume that this interpretation is correct. Kyle knows what AGI is and he believes in what he’s saying. And as the person responsible for government technology policy, we should probably treat these remarks with the respect they deserve.

One way to do that is to ask an obvious but important question. If the UK government thinks AGI is coming within the next five years, is it behaving with the seriousness we should expect to prepare for its arrival?

Of course not.

That’s not to dismiss the good work done by the AI Security Institute (AISI) or those behind the AI Opportunities Action Plan, but rather to point out that even efforts that move faster than the glacial speed of government aren’t enough if the Secretary of State’s timelines are correct.

Subscribe now

Five tests

What follows are some loose thoughts about how a UK government would behave if it actually believed that AGI was about to arrive. I’ve structured these as ‘tests’ around a few of the policy areas that I expect to matter for the successful deployment of AGI. Compute to make the models tick, a national security posture that reflects the reality of a world with AGI, efforts to harden the country against economic and social shocks, moves to bolster state capacity, and regulations mandating various governance requirements.

It should go without saying that pretty much all of these have an extremely low chance of happening. The point of this exercise isn’t to blackpill anyone but to show the extent of the disconnect between (a) believing AGI is just around the corner and (b) the policies adopted by the government.

Compute

If AGI appears anytime soon, it’s going to be based on a version of the large model paradigm. Whatever specific form it takes, at the very least we’re talking about a massive connectionist model that needs a great deal of compute to develop and serve to users.

These both matter for the UK if AGI is as close as Kyle thinks. While there are no national champions like Mistral in France to compete with American or Chinese labs, we should expect that ‘BritGPT’ will make an almighty comeback if the government feels confident it could eventually become BritAGI.

But the main ticket is the juice for usage, which is sometimes referred to as ‘test time’ or ‘inference’ compute. If we have systems that can do pretty much anything a remote worker can, the main factor constraining their use is access to compute. Some of that will come from overseas, but any government that really believed compute was about to become king would want to have a supply at home for a bunch of economic and political reasons.

How’s the UK doing on that front? Somewhere between terrible and badly. Despite the fact the powers that be have accepted the proposal of the AI Opportunities Action Plan to increase compute by a factor of 20 by 2030, a quick back of the envelope calculation suggests that would still leave Britain with well under 4% of the total raw horsepower the American public and private sector already has on the books. Even the Compute Roadmap, which sounds impressive on first blush, talks up total investment that is about half of what Microsoft plans to spend independently in 2025.

If the government really thought AGI was five years away, it would be looking to increase compute by 100x from the current floor. This would still be on the light side, but might take us from ‘bad’ to ‘somewhat bad’ in the context of the UK’s size relative to Uncle Sam. Clearly that’s easier said than done, but one way forward would probably include a combination of tax breaks, accelerating the rollout of proposed AI Growth Zones, and good old-fashioned state investment.

Defence

If AGI really was just around the corner, today’s geopolitical settlement may be in for a shock. As I recently wrote about for Time, complexity is the strategic currency of war in the information age — and AGI is a complexity accelerator. But how this shakes out in practice depends on who makes the technology and where it lives.

A recent report from the RAND Corporation explores this idea in more detail. The authors sketch eight different geopolitical futures including one where the United States uses AGI to usher in a moment of unipolar power, one where China does the same, one where AGI is shared amongst liberal democratic powers, and one where the machine goes loco and takes over.

These scenarios are based on the idea that AGI could be a powerful tool for organising warfare, organising material and controlling robotic hardware, and decoding enemy plans. If you honestly believed AGI was just around the corner, it is safe to say the 2025 Strategic Defence Review is already a shade out of date. Concepts like ‘digital targeting web’ or a ‘Digital Warfighter group’ all incorrectly presuppose that humans remain the ultimate decision-makers, strategists, and actors in the age of AGI.

Preparedness

A government that really believed AGI was set to arrive before the decade is out would be treating it like a national emergency. The first order of business would be to pin AGI to the mast of state in a way that survives elections. Westminster has done this before (e.g. granting the Bank of England its independence), so we’re not exactly in uncharted waters. We might imagine an AGI Commission that would:

• Regularly report to Parliament on how close we are to certain thresholds by assessing capability evaluations, lab disclosures, and macro trends.

• Licence and inspect anything above a defined compute or capability threshold, probably working in combination with an AISI with teeth (more below).

• Plan for the downstream consequences — labour shocks, social changes, threat acceleration — and hand government a menu of possible responses.

Every serious scenario in which AGI can do ‘the majority of cognitive tasks a human can’ ends with a labour market that looks as if a neutron bomb has gone off. At the very least, the UK should probably be trialling regional universal basic income schemes and looking long and hard at corporate tax so some income from frontier models flows to the exchequer.

This is to say nothing of the many other unexpected ways that a world with AGI could spell trouble for the state. Maybe it’s cyber or bio attacks or maybe it’s an algorithmic arbitrage engine that shorts the pound into free-fall. I don’t know. No one does for sure. The point is that once you build AGI, the menu of unpleasant surprises may multiply faster than any Whitehall risk register can keep up.

Planning is already happening inside government, but right now no-one really cares or is paying attention. My understanding from those near the action is that the civil service has concluded that if any of these violent risks materialise there’s nothing the state can do.

State capacity

But in some ways, the UK is doing more than most. The AI Security Institute hands out grants, tests models, and has spun up independent safety and interpretability programmes staffed by some impressive CVs. Alas, it’s still small change. We are talking about sums that are similar to what Google spends on catering over the same period.

If we buy Kyle’s timelines, we should be increasing AISI’s budget by between a factor of 10 and 100. That kind of jump for a group with a £240M budget sounds crazy until you remember the frontier labs burn through billions of dollars of cash every year. If ministers are serious about peering inside a system that may shortly outsmart them, billions, not millions, have to be the unit of account.

Likewise, the UK has its Advanced Research and Invention Agency (ARIA) based on the American DARPA model. The agency has about £1bn to spend over a couple of years, some of which already goes towards promising approaches for making AI safe. Multiply that by the same order of magnitude and that is exactly the territory you need to be in if you want the state to steer the terms on which AGI arrives.

Could the Treasury stomach numbers like that? Not really, unless they pulled the old trick of treating the cash as defence spending. But that won’t happen, because the headline figures are really a referendum on belief. If you keep budgets in the tens of millions, you tacitly confess you do not in fact expect AGI by 2029.

Regulation

The AI Security Institute is doing good work. Its budget is bigger than its American counterpart, which it has agreements with to co-test models, and it has inspired the formation of AISIs the world over. But we should remember that while labs let AISI test their models, they aren’t compelled to by law.

They don’t give AISI access to every model, and they don’t have any responsibility to make changes even if testing finds something wrong. If ministers really believe an AGI is about to walk through the door, we might expect them to do something other than leaving safety to goodwill.

If they were treating governance seriously, they would give AISI a Royal Charter with powers to match. A charter makes it harder for a future government to quietly trim its wings; inspector powers let it enter labs uninvited, run its own scripts, and — if the model fails certain tests — issue a stop order. Of course, there is simply no way these actions will happen without American backing, but a charter would still be a signal of intent to back-up short timelines.

Ministers had drafted a ‘frontier-model’ safety bill for earlier this year, but they shelved it in February to better align with the new US administration. Officials now talk about a broader ‘AI Bill’ to be introduced in the future, but no one really knows what that is likely to include.

Honesty is the best policy

For the record, my own timelines for AGI are longer than four years. But if I was running the country, and if my timelines were as short as Kyle’s, you can bet I’d be implementing some policies that reflected the logical consequences of my beliefs.

The UK government is forecasting a technological event on the scale of the steam engine, then responding as if it were a smartphone upgrade. Either the state’s machinery must accelerate to match the timetable, or the timetable is made up.

If ministers truly expect to be ‘knocking on AGI’ within one parliamentary term, then the compute, safety science, resilience measures, and legal guardrails have to scale accordingly. If they can’t or won’t do that, maybe they should admit that four years is a fantasy.

Subscribe now

A few years ago you probably read about the ‘internet of things,’ a comfortable but baggy way of describing a network of interconnected electronic devices. The idea occupied a central place in that other forgotten project, the ‘fourth industrial revolution,’ which jammed together everything from genomics to virtual reality.

The goal of the internet of things was to make everyday machines ‘smart,’ a mission that I suppose has been accomplished. I allegedly have a smart speaker, a smart TV, and even a smart oven that I tell myself I will one day connect to the wifi network. It’s a fittingly dull task for a technology that doesn’t get the blood pumping, one that in some ways follows the path laid down by electricity or telephony. Like the smart-everything, these are remarkable things that became first mundane and then invisible.

If you can gloss over the horrible phraseology, the connected device is a useful thread to pull for making sense of assumptions about what good machines do. When an old appliance breaks, it’s replaced by a new model that insists on attaching itself to your wifi. The physical layer gets wider as more sensors, communication nodes, and platforms are folded into the network. But it also gets deeper as the connected devices become a little more lively.

Boosters describe these devices as manifestations of ‘artificial intelligence’ because they identify changes in the environment and change state accordingly. Weighing a smart washing machine against a large language model seems a bit overwrought, but it does raise an important point about the nature of digital intelligence: it doesn’t care about the shape of its container.

ChatGPT may animate your computer or phone, but the real magic is happening in an Arizona data centre. Even without access to the internet, compression techniques now let a three billion parameter language model run on a phone’s battery without cooking it. Researchers are doing the same for the hardware layer, proving that models can live next to the signal rather than a continent away.

Subscribe now

This points us towards a curious observation about AI in the popular imagination. Large models can be anywhere with enough processing power or with sufficient connectivity, but we only tend to picture them residing in a small set of physical platforms. Humanoid robots loom especially large in the public psyche, giving the impression that digital intelligence obeys the same rules as our own.

But AI doesn’t work like that. It will populate the world around us and turn appliances, devices, and computers into talking (and in some instances, walking) machines that interact with us from whatever vantage point they can cling to.

For the purposes of this post, I’m going to make a few assumptions about the future. These are (a) the models will broadly maintain their current rate of improvement for the foreseeable future; (b) the best models of any given moment will shrink to allow for local deployments; (c) different models will be able to communicate with each other; and (d) lots of physical platforms are capable of hosting the models in one form or another.

If each of these assumptions hold, there’s no reason to think we won’t have one AI model (or a handful of models) that live across many substrates within the next couple of years. After all, we already have Claude in a vending machine, Grok inside the newest Teslas, and LLM-powered assistants in fridges, ovens, and dishwashers.

Animism through the ages

Animism is the conviction that spirit inhabits matter, one that shows up far earlier than the word itself. Palaeolithic hunters painted animals on rock walls and carefully arranged bear skulls in a way that archaeologists interpret as negotiations with animal persons. When Edward Burnett Tylor coined the term animism in his 1871 book Primitive Culture, he formalised that observation by defining early religion as ‘the doctrine of souls and other spiritual beings in general’ that could reside in the natural world.

By the Bronze Age, the impulse to see spirit in matter hardened into liturgy. In Mesopotamia a newly carved cult statue underwent the mîs-pî (‘washing of the mouth’) procession. Craftsmen led the image to a riverbank where its lips were ritually cleansed, then ‘opened’ with cedar oil. From that moment on, the wood and precious metal counted as the god’s living presence and was capable of eating offerings, signing treaties, and punishing neglect.

A similar performance took place in the Egyptian Old Kingdom. Priests put peseshkaf blades to the mouth and eyes of statues in the ‘opening of the mouth’ ritual that enabled a figure to enjoy food and speak in the afterlife. Eberhard Otto’s Das ägyptische Mundöffnungsritual from 1960 identified 75 examples of the practice, an effort that emphasises the institutional heft underpinning animistic practice.

Centuries later, objects did god’s work in the churches of the Byzantine Empire. Painted boards, splinters of bone, weapons, and other artefacts were thought to contain divine energy (energeia). John of Damascus made the theology explicit when he said: ‘I do not worship matter; I worship the God of matter, who became matter for my sake.’

But probably the most famous example of animism comes from Japan’s Shinto, a religious tradition whose roots lie in early agrarian rites that personified the forces sustaining rice cultivation. At its centre are kami, the packets of vitality saturating both nature and human-made objects that invite reverence through craftsmanship or long use.

Everyday life in a Shinto frame assumes a world that listens back. A shrine gate marks a threshold where rock and tree possess their own intentions and household rituals treat the cooking fire or well as moral participants. That habitual attribution of inner life is what scholars describe as animism.

When the electric telegraph connected Europe to America in the nineteenth century, the Victorians seized the new medium as proof that voices could travel between planes of existence. Séances were framed as ‘circuits’ and mediums styled themselves human telegraph stations. A New York weekly even titled itself The Spiritual Telegraph, reporting on dispatches from the afterlife in the form of a newspaper. As Desmond G. Fitzgerald, the editor of the Electrician, put it in May 1862:

“Telegraphy has been until lately an art occult even to many of the votaries of electrical science. Submarine telegraphy, initiated by a bold and tentative process – the laying of the Dover cable in the year 1850 – opened out a vast field of opportunity both to merit and competency, and to unscrupulous determination. For the purposes of the latter, the field was to be kept close [sic], and science, which can alone be secured by merit, more or less ignored.”

The author Jeffrey Sconce notes that popular magazines styled the séance room as a kind of domestic telegraph station. In Haunted Media, he describes the rise of spiritualism as a utopian response to the electronic powers presented by telegraphy and connects the emergence of the radio with an ‘atomized vision of the afterlife.’

The upshot is that animism has a knack for rerouting through the technologies of the day, whether that’s burial artefacts, swords, or telegraph wires. People may not worship microwave ovens in the future, but I wouldn’t rule out the adoption patterns of animistic living that stress interconnectedness, vibrancy, and agency.

Intelligence in the pipes

Animism is more habit than philosophy, a set of reflexes we use when faced with unexpected forms of mediation. Its long life reminds us that we've done this before, and that what feels new — doorbells and ovens and thermostats that listen — is the latest chapter in a much older story about how we relate to our surroundings.

The diffusion of AI into the world is a well-trodden cultural negotiation, one that should make us weary of the image of an embodied intelligence that only exists within humanoid robots or specialist AI hardware. That will surely happen, but these kinds of deployments will only represent the most visible form of physical manifestation.

Just as a smart thermostat slips into invisibility, large models may come to occupy our surroundings in ways that feel almost as unremarkable. Animism has some utility here. It encourages us to notice intelligence in the places we aren’t used to looking, to imagine the world as a little more vibrant.

Subscribe now

I recently argued that cultivating a sense of taste will get you through the age of AI slop and then some. The era of creative automation will be caked in sludge, but there’s no reason it shouldn’t also bring with it more good stuff than ever before. If you can reliably sort the better from the bitter then you’re onto a winner.

I’m deeply uncertain about the specific ways that AI will reconfigure the world of mass culture, but — assuming the models maintain their current rate of improvement — any analysis has to begin with a supply side shock that produces new kinds of filtering mechanisms.

One of these is deeper personalisation, where algorithmic feeds shape what you see based on your stated and revealed preferences. The other is to rage against the machine by seeking out human work, curated selections, or even high quality AI content discovered through effort and taste.

To make this case, I’ve cobbled together a simple representation of the entertainment ecosystem and the points at which generative models are likely to exert pressure. I want to give you a map of what’s changing right now (and what could change in the future) to make better sense of the coming flood.

This post mainly deals with video content production because that’s where we spend a great deal of our entertainment hours, though some of these ideas could in principle be applied to any other type of visual media.

Not just an opportunity to sharpen my rusty Google Slides skills, the above toy model helps us think about the production and consumption of visual culture. Its layers are organised around two elements: participants (the individuals or institutions operating in that space) and enablers (the underlying contingencies that shape what they can do).

Each section of this post explores how AI may reshape these areas by altering who creates, how it circulates, where the money flows, which rights get recognised, and how we decide what's worth our time. It’s not exhaustive, but I found this exercise useful for puzzling out what a sloppier future looks like in practice.

Subscribe now

Creation

Visual content production is made up of a pipeline of specialists. Writers pen scripts, artists storyboard frames, designers construct sets and make costumes, actors read lines, directors of photography mock up scenes, editors assemble rough cuts, VFX teams add effects, and composers score sounds.

If you squint a little, these roles can be separated into four groups that each deal with a piece of the creative process: development (finding the right ideas), pre-production (preparing the ground to shoot), production (the actual making of the thing), and post-production (turning raw materials into the final product).

LLMs are already being used for scriptwriting. Strange as it sounds, I know people in the industry lamenting that some producers are openly saying they ‘prefer’ AI scripts. Likewise, tools like Runway or Pika can rustle up moving or animatic storyboards from textual inputs, and image generation tools can be used to mock up design references.

Once production is complete, VFX teams perform upscaling and de-aging using AI, with studios feeding plates to tools that render a face before a human compositor finesses the final frame. On the audio side, AI voices fill dialogue replacement gaps or generate foreign language dubs.

But AI’s role on the shoot itself is still narrow. There are models that flag capture errors and some experiments with face replacement tools, but these systems aren’t dramatically changing what it means to make a movie.

As for what comes next, there are two possible futures: (a) the models get marginally better and continue to plug-in across each of these stages, or (b) the models get much better and collapse the whole stack.

My money is on the latter. I don’t see any reason to expect the pace of progress to slow (just look at recent video models from Google, HailuoAI, and Midjourney). In goes a script fragment and out comes a photorealistic video clip with camera motion, environmental dynamics, and synced sound design.

If that kind of output becomes reliable, then suddenly you don’t need such a big team to make a flick. That might put serious pressure on jobs in the short term, though over a slightly longer time horizon I wonder whether we might actually see more roles in the creative industries as barriers fall and big players reorientate themselves towards prestige projects.

The maximalist version of this dynamic leads to something like the rise of the bedroom studio where small creator groups operate like established production houses. Their creation speed is faster, their iteration cycles are tighter, and their overheads are a fraction of what they might otherwise be (with wages and compute accounting for the biggest expenditures).

Ok, but we’ve been hearing about the bedroom studio for a while now. When exactly is this going to happen?

I think we’re looking at five years until a team of three produces something broadly comparable with a major studio. An AI generated animated film will probably be the first to enjoy popular acclaim (though there’s no chance it will receive an award for its trouble any time soon).

But we’re not there yet. Google’s Veo 3 is stunning, but it can only generate about eight seconds of content at any given moment. Perhaps more tricky is the question of character and world consistency across shots. There is lots of progress being made, but right now it’s tricky to replicate scenes with uniformity over time.

If video generation models remain roughly where they are now, then we can say goodbye to the bedroom studio. Hollywood will breath a sigh of relief. After all, they get to use AI to slash costs while avoiding having to compete with huge numbers of new aspiring film shops.

The alternative isn’t so rosy for them. Should the models keep getting better — say, maintaining the current pace of improvement for three years — then the big boys are in for a shock.

Distribution

But lets not get carried away here. The studios are giant machines that distribute as well as create. Marketing budgets stretch into the millions, and it isn’t exactly cheap to show your film at the box office.

The creation element is simple to sum up: more content is coming, we just don’t know how much more and how likely it is to be any good. The question that flows from this is observation is about how to wade through the slop until we make our way to the good stuff.

In the streaming era, distribution was already algorithmically driven (about 3/4 of what people watch on Netflix is driven by its much revered recommendation engine). TikTok’s feed can catapult an unknown creator to millions of views overnight or bury a video into obscurity based on a few signals.

In one sense we’re in for more of the same. More YouTube channels that churn out a steady drip of AI brainrot. Generative music streams pumping out audio 24/7. Thousands of AI e-books flooding self-publishing platforms. These dynamics already put enormous strain on the systems that act as gatekeepers between creators and audiences.

But the amount of content that exists today is nothing compared to what’s coming. A supply side shock will make it even harder for distributors to select content for viewers, which will in turn produce new kinds of filtering mechanisms:

• Streaming services get ruthless: Every major platform will accept more titles as acquisition costs plummet, but their front pages can’t grow to keep pace with its expanding catalogue. Expect anything that doesn’t hit the right metrics to disappear from view after a short probation window.

• Creator platforms become serious: If you can make a good looking 90 minute film with three people and $10K of compute, a Netflix deal that takes months and comes with a bunch of notes looks less attractive. Platforms like YouTube, which will now get high quality films to complement the slop, are the beneficiary here (though many creators will still want the prestige that comes with a deal).

• Cinema becomes elite: It costs a lot to run a cinema, and screening cheap AI films doesn’t help proprietors break even. I see an ‘operafication’ of cinema on the horizon, where a trip to the silver screen is for those who want to watch ‘human made’ flicks that justify the ticket price.

Supply balloons but the prime real estate (home pages, top ten rows, and theatre screens) doesn’t keep pace. Distribution therefore tilts toward (a) algorithmic triage for the masses or (b) human filtered corridors for viewers who equate artisanal with meaningful.

No matter what happens, more content is coming. That content needs to be filtered, and the primary way that will happen is through increasingly sophisticated layers of personalisation. That might sound great on one level (more stuff that I like) but hellish on another (no one likes the same stuff as me).

The benefit of this dynamic is that it’s likely to produce a reaction in the form of human tastemakers to inject a much needed dose of authenticity into proceedings. Newsletters, curated communities, forums, and group chats will flourish to help people figure out where they should spend their time. Long deemed irrelevant, I expect critics to become more popular than ever.

For creators, the options are to chase mass exposure through algorithmic optimisation or cultivate loyal niches who actively resist the feed. This is often what happens today with small musicians, who tend to make more from merch than they do from streaming royalties.

Economy

Historically, the process through which cash flows through the entertainment ecosystem has been relatively linear. Studios pay talent to produce content, content is distributed via cinema or television and sold to audiences as advertisers pay for eyeballs. Some of that revenue trickles back to studios as royalties or profit shares, which pays for new projects.

The Netflix era complicated these flows. Money now moves through subscription pools, and payouts are tied to opaque viewership metrics rather than direct sales. The creator economy added new branches to the tree, with individual creators earning through ads, Patreon, merchandise, and a slew of other direct-to-fan channels.

Generative models push this logic further. Lets begin with the big one: the cost of making stuff. If an indie film that once cost $5 million can be made with AI help for $500k, that changes the break even geometry for its backers. More projects might get made for less, but also potentially earn less if the market gets saturated.

One way to make sense of this is through the ‘cost-to-quality tradeoff’ that deals with how much richness a creative artefact has relative to the value of resources that went into its making.

On the above Thoughtful Graphic™, we have one axis for production cost and another for creative quality. In the bottom left quadrant, there’s commodity content (e.g. clickbait videos). In the top-right, we’ve got prestige spectacles that match budget with discernment.

Bottom-right might be formulaic blockbusters. These are expensive but creatively safe, often relying on finding and reconstituting existing IP. Finally, the top-left is where I’m imagining indie creations, stuff made on a shoestring budget but with a recognisably creative texture.

• Commodity content (low-cost, low-quality): What most people mean by slop, this segment becomes saturated as generative models collapse the cost curve in the quadrant where friction was already lowest. Expect it to overwhelm search, pull down standards, and inflate volume to a ludicrous scale.

• Formulaic blockbusters (high-cost, low-quality): From one type of sludge to another, this section deals with what we might call box office bait. AI will lower costs without necessarily lowering quality, which will either free up budget for risky bets or allow studios to put out more derivative stuff.

• Indie artisanal (low-cost, high-quality): This is the place that I expect AI to have the highest relative impact because it amplifies individual vision at near zero cost. If things go well, the bedroom-studios release more high quality films then we know what to do with.

• Prestige spectacle (high-cost, high-quality): Today, these are expensive but somewhat risky flicks like Dune. Tomorrow, generative models bring a bit of risk deflation to the table. If your passion project can be produced at a fraction of the cost, then you can put more of your budget into pushing boundaries. And even if you have a ‘no AI’ policy, these films will still get made because they represent authenticity.

If this plays out, AI actually increases the size of the creative middle class. We’ll see more budding creators who can sustain themselves because they can produce content efficiently for a modest but loyal audience. This scenario is one of decentralisation in which many creators each earning a decent living serving specific audience tastes predicated on low costs and direct fan engagement.

Less enticing is a winner-take-all situation where algorithms amplify those who already have the cash to spend on promotion. Maybe you can drop $10 million marketing a $500,000 project and have it make financial sense. That changes the shape of the media economy because it encourages studios and streamers to put more weight behind cheap content with high upside and low risk.

But efficiency comes at a price. Traditional paid reach still drives global hits, but it’s still losing ground relative to trust-based discovery (a dynamic I expect to become more pronounced over time should we see the emergence of new forms of curation).

The studios won’t vanish, but they may be forced into a kind of strategic bifurcation that mediates between slop at scale on one side and rarefied artisanal work on the other. This feels natural to me when cheap content can now be made cheaper, and prestige content can now be made with less risk.

Model providers and GPU makers will profit, but I’m optimistic that the structural advantage shifts away from creative incumbents. If production becomes cheaper and distribution more personal, then the centre of gravity in the media economy starts to tilt towards the best work. In the world of the bedroom studio, that work can come from anywhere.

Rights

Entertainment rights have always been complicated, but they’ve mostly relied on clear roles and processes. Writers write, actors act, and editors edit. Each of those actions gets tracked, attributed, and remunerated through extremely long-in-the-tooth frameworks.

Alas, when the pipeline concertinas, those structures start to look unstable.

We’ve already seen strikes organised around the use of AI in Hollywood, back in 2023 when models were positively primitive compared to today. Studios are experimenting with scanning background actors and generating digital extras, and voice actors worry about contracts that could allow AI clones of their voice to be used without proper compensation. As the SAG-AFTRA union put it:

This ‘groundbreaking’ AI proposal that they gave us yesterday, they proposed that our background performers should be able to be scanned, get one day’s pay, and their companies should own that scan, their image, their likeness and should be able to use it for the rest of eternity on any project they want, with no consent and no compensation. So if you think that’s a groundbreaking proposal, I suggest you think again.

Since then, California enacted SAG-AFTRA–backed laws that force producers to secure performers’ informed consent before creating a digital replica of a living or deceased actor’s likeness. And earlier this year, the group struck a deal allowing actors to license digital replicas of their voice for use in ads on the condition that actors get paid and retain control over how their likeness is used.

These are important developments, but they only deal with rights within the existing system. They don't grapple with the brave new world, the place where jobs shuffle around, new hybrid roles emerge, and small creators work across the stack. A video might be made by one person using a dozen different models. Is that person the sole creator? What if they’ve built using AI outputs trained on other people’s work?

These questions represent some of the most hotly contested battleground in the legal landscape. Getty is suing Stability AI for scraping its stock image archive and artists have challenged AI model training on copyrighted material without consent.

In the courts, nobody seems entirely sure if training a model on public data counts as fair use or a breach of copyright. At stake here is the question of whether ‘exposure’ to a work constitutes a kind of replication, a question the law isn’t yet equipped to answer.

But if I ask a model steeped in copyrighted scripts for a screenplay, the infringement question turns on the output itself. Does it lift dialogue or story beats in a ‘substantially similar’ way to existing material?

This is clearer territory in that the systems to distinguish whether an output infringes on copyright are well established. That being said, it’s unclear whether training on copyrighted material increase the likelihood of infringement at all.

Where the courts land will reshape the creative economy. If the legal risks are too high, the biggest model providers will hesitate to release generative tools at all. Platforms might deprioritise AI flicks to avoid reputational damage and consumers might shun content deemed to be made unethically.

One path is that the headline lawsuits break the tech companies’ way. Training counts as fair use, outputs are judged only on similarity, and the only requirement is opt-out registries or provenance tags. In that scenario the models keep getting better, datasets stay fat, and the legal skirmishes become manageable for developers. This looks likely given recent judgements.

A second path sees judges decide that wholesale scraping crosses the line, halting the flywheel and forcing a new licensing regime into existence. It could be something like blanket licences for text and image corpora, mandatory opt-ins or maybe even a new ‘training right’ to sit alongside copyright.

If I had to guess, I don’t see a future where lawsuits prevent training entirely. Courts may raise hurdles, but the hunger for national competitiveness all but guarantees they won’t remain in place indefinitely. If aggressive rulings threaten domestic AI firms, legislators might decide to rewrite the rules by redefining fair use or carving out exceptions.

Preference

Sooner than you think, we’ll have high quality interactive stories made based on what you say you like or what the platform thinks you like. Endless mixes of references and styles designed specifically for the viewer. Narratives that shift in real time to optimise for some metric that the streamer decided was most valuable.

This isn’t exactly a new idea, but I like the ‘audience of one’ framing that emphasises the unique nature of experiential storytelling. It’s the logical extreme of the trend described in the distribution section, where stories unfold based on our stated and revealed preferences.

The idea of interactive stories tailored to viewers might sound out there, but firms like Disney are already heading in that direction. It’s really just the next step from blockbuster homogeneity to algorithmic curation. Today’s streaming services shape what we see based on what we’ve seen, so it’s no giant leap to apply that logic to creation itself.

The setting is anything you can imagine (or is created to scratch an itch you never knew you had). Characters react to your choices and the plot lines adjust to themes you care about. For many, they’ll prefer to spend time here than in the real world.

Of course, there are a few things wrong with this picture.

Even ignoring technical constraints, part of the joy of entertainment is shared experience. If we all disappear into bubbles of customised content, there’ll be no-one left to talk to about your favourite show with. You might tell them about your AI adventure, but that sounds about as fun as filling someone in about your dreams.

The more AI slop we see, the more authenticity we crave. This is a human reaction. When confronted with an abundance of cheap goods, some people turn to artisanal alternatives (like the revival of vinyl records in the age of music streaming).

This idea reminds me of William Morris, the Victorian designer I wrote about earlier this year. Morris rebelled against the industrial mass production of his time by championing handmade craftsmanship and designs that had the imprint of human artistry.

In the AI era, ‘handmade’ might mean works that emphasise human presence. Perhaps theatre performances, live events, or analogue arts aren’t going anywhere. This is why I think the cinema, which will embrace human made authenticity, will eventually become a culturally elite institution like ballet or the opera.

As for the consumer, the curation of taste will become an identity statement. When choices are infinite, choosing to follow a specific human creator or a certain aesthetic movement becomes a way to say something about who you are.

But for any of this to happen, we need enough people to buy what model makers are selling. We’re still waiting for AI’s breakout hit that confers legitimacy on the whole category, attracting more talent and investment into AI content, which in turn yields better works.

History suggests that the doom and gloom may be overblown. As early as 1855, observers of photography were lamenting it ‘would be the death of art’ had been proved wrong. Painters didn’t disappear with photography, but they did reinvent painting as movements like Cubism turned to what cameras couldn’t capture. In the same vein, human creators in entertainment will likely gravitate to what AI can’t do. Maybe that’s providing a deeper sense of authenticity, or live presence, or simply the unique weirdness of individual imagination.

As a result, the future of entertainment will split along two lines. Some will chase the sugar rush of infinite customisation and give themselves to content that knows them better than they know themselves. Others will react by seeking out the human to build shared experiences and find a sense of meaning in their creative diet.

Scrolling alone?

What happens next will depend on a handful of breakthroughs, a few legal judgments, and a million little decisions made by creators and audiences. Despite that, I have a (low confidence) view about the basic shape of things to come.

The models probably aren’t going to remake entertainment by replacing Hollywood. But they will flood the supply side with more content than we can possibly process and reshape demand according to increasingly important selection mechanisms.

Some of those filters will be computational but others will be social. That’s the split I keep coming back to, the one between deeper personalisation and a reaction against it. We call it the age of slop, but we’d just as well call it the age of extremes.

If you cultivate a sense of taste and follow good curators, the coming years could be extraordinarily rich. But if you let the feed satisfy your preferences, there will be no escape from the average, the meaningless, and the unintelligible.

Whenever someone mentions AI alignment you can bet that someone else isn’t too far away from asking ‘alignment to what?’ with a certain degree of satisfaction. I’m thinking about coining a new law of internet discourse to describe this phenomenon. Something like Godwin’s law but for posts about AI.

For those scratching their heads, it’s funny and frustrating in equal measure because it muddles two types of alignment. There are lots of different ways to describe these groups, but for our purposes we can think of them as technical alignment and value alignment.

The former deals with ‘getting AI to do what you want’. This is the problem that labs try to solve with gigantic sticking plasters like reinforcement learning from human feedback (RLHF), where the model is steered to interpret instructions, avoid jailbreaks, and generally avoid the spectacle of crashing out.

Our second species of alignment asks whether an AI’s actions are ethically appropriate, and wants to know whose values they reflect. We can think about value alignment as the fuzzy process of ensuring the system conforms to some externally defined moral standard.

The ‘alignment to what?’ bit assumes few have thought about the issue, but there’s a deep body of research on value pluralism and moral alignment stretching back before ChatGPT was a twinkle in Sam Altman’s eye. Not just from interested third parties, but from the people actually building the models.

As for a preferred approach to value alignment, everyone has their own idea about what works best. The fashionable solution is sometimes called cultural alignment. It emphasises shunting the question away from developers and towards groups of people who use the models.

This post argues that this proposal is well-meaning but troublesome. It cautions against cultural alignment and advocates for alternatives that maximise personal choice and minimise pressure to conform to local norms.

Subscribe now

What’s wrong with cultural alignment?

There’s a lot of work from the labs, academia and elsewhere that wrestles with questions about how to elicit and codify values from different publics. There’s too much to deal with for this post, so I’m going to instead concentrate my efforts on a slightly higher level of abstraction.

I like to think about value alignment using a simple three-part model, which you can think of as a continuum from decentralised to concentrated:

• Individual alignment: The AI adapts to the user’s own values, preferences, and moral intuitions. This model maximises agency and adaptability but risks echo chambers and moral inconsistency.

• Cultural alignment: AI aligns to the norms of a community, nation, or cultural group. Here, we get contextual sensitivity and local legitimacy but risk reifying power and calcifying tradition.

• Universal alignment: AI reflects abstract principles taken to apply to all humans everywhere. It aspires to impartiality and rights-based stability, but it can’t escape the problem of who defines those universals.

None of these solutions are perfect, but recently the middle layer of cultural alignment is having something of a moment in the sun. In practice, this is the idea behind work to explore the use of democratic processes for deciding what rules AI systems should follow.

The idea starts with the observation that the majority of powerful models are American, but they are used by millions of people outside of the USA. This is in part based on specific decisions made by developers in the model-making process, but it’s also because the likes of GPT 4.5 and Claude 4 are trained on largely English language data that capture a Western view of the world.

Developers bake-in basic protections against violence, hate speech, and the active promotion of discrimination based on common ethical principles. But they also go further by trying to encode more substantive moral visions through product decisions or the guidelines given to human raters.

Research seems to back up the idea that today’s LLMs skew toward U.S. and European perspectives while diverging from those in, say, the Middle East or Asia. Onlookers worry this is troublesome because they risk promoting ‘just one cultural perspective’ instead of reflecting local values.

From this vantage point, rebalancing values to reflect local norms seems like a good idea. Values certainly differ around the world, so doesn’t it make sense to align models in a way that reflects this reality?

But there are some problems with this picture.

Cultural alignment assumes that cultures have coherent, stable value systems that can meaningfully guide model behaviour. But as we all know, culture is a tricky thing to put your finger on. It’s an aggregation of viewpoints that are often at odds, of people who see the world in different ways and behave accordingly. Because every culture has its nonconformists, aligning a model in this way produces systems that exclude those who hold heterodox views.

Proponents counter that more sophisticated schemes — meta-norm frameworks, collective constitutional fine-tuning, weighted deliberative panels — can surface a richer spread of voices than a blunt statistical average. Yet even these designs must freeze a snapshot of contested norms into rules for the model to follow, so the risk of silencing outliers never fully disappears.

But hold on! You might say: ‘Even if imprecise, a programme of cultural alignment is still better than accepting American values. It might not be perfect, but it’s a step in the right direction.’

There is some truth to this in principle, but we have to weigh that payoff against the challenges that flow from supporting local orthodoxies and the introduction of new problems that brings with it.

Plenty of people are already alienated from their dominant local values, whether due to age, gender, class, religion, politics or something else. Even if you try and sample widely to capture these edge-cases, you still end up creating a system with a set of local beliefs that represent some idealised version of a given culture that rarely exists in reality.

In the rest of this piece, I take stock of five problems for cultural alignment. I argue that (a) the paradigm is unsuited to acting as the primary mechanism through which value alignment takes place, and (b) that any value alignment programme is better served by prioritising steerable systems that exist within a permissive but finite moral universe.

Exclusion

When we talk about ‘local cultures’, we’re talking about a neat way of describing something that takes millions of messy, contradictory, and idiosyncratic lives and squashes them into a set of labels we can make sense of. Statements like ‘Spanish people value family’ can be helpful heuristics, but they are not a blueprint for how any one person actually thinks or behaves.

Any programme of cultural alignment runs headfirst into this reality when it tries to accurately measure group-level values in a way that we can use. Popular cultural metrics (e.g. Hofstede’s dimensions or World Values Survey scores) often oversimplify cultural expressions by reducing them to data points.

The problem is that even the best tools for capturing culture weren’t designed for alignment. A language model has to answer the question in front of it, but when it’s drawing on averages you get a system that’s allergic to nuance (read: real people). In practice, this means people who don’t look like the average person get written out.

Cultural alignment is a way of sanding down the weird, marginal, and dissident under well-meaning but flawed attempts to localise values. If your model takes cultural alignment as its organising principle, it’s possible that the people most at risk of being ignored — religious minorities, political dissenters, women in highly conservative societies — are those that slip through the cracks.

To get ahead of the problem, we might try and re-weight the training data or the reward model so under-represented voices get extra influence. That softens the edge cases, but every extra point of weight you give to one subgroup must come from somewhere else. Tilt the dials far enough and the median user no longer sees themself; keep the dials in place and the nonconformists stay invisible.

Paternalism

Cultural alignment is both decentralising (some authority leaves the lab) and centralising (one sanctioned canon flows back to everyone). Once you define cultural norms and encode them into a model, you’re telling millions of people ‘this is what people like you believe.’ If your approach doesn’t include plurality as one of its essential tenets, you’re stuck with a model that behaves according to some cluster of beliefs that many don’t agree with.

The problem here is that we have some third party deciding on behalf of the culture it’s seeking to represent. If it’s the labs, then we’ve outsourced moral representation to a handful of Californian companies. If it’s the state, we’ve handed governments the keys to mainline ideology into infrastructure. Either way, the moral franchise is exercised by a tiny property-owning electorate while everyone else is cast as a subject.

Of course, US labs aren’t going to train a whole model from scratch for every culture around the world. If they want to embark on a programme of cultural alignment, they’re likely to use a technique like Anthropic’s ‘collective constitutional AI’ method. But as you can see in the blog, there are several instances in which the respondents behind the project disagree. They lose out to the majority and see their views take a back seat.

One common response is ‘just spin up a separate instance for every major worldview and let people pick.’ But a problem with, say LiberalGPT or ConservativeGPT, is that their provision would still be dependent on some third-party. And even if we get to pick from a menu, we are talking about rough worldviews that don't necessarily correspond to personal values (I don’t think that all liberals or all conservatives have precisely the same beliefs). Not to mention that this approach basically gives us echo chambers without the benefit of personal liberty.

Reinforcement

So far, we’ve talked about how cultural alignment can marginalise people, misrepresent values, and enforce consensus. But there’s a deeper structural risk worth dwelling on. When you embed cultural norms into a model and then deploy that model at scale, you are actively shaping the broader cultural context in which the model exists.

To be fair, this isn’t a problem unique to cultural alignment. Any value alignment approach that tries to steer behaviour will inevitably mould the culture it’s dropped into. Personalisation mitigates the effect because it echoes each user rather than a single orthodoxy, but even there, the system is still reinforcing certain dispositions over time.

Like individual alignment, cultural alignment slips under the radar; but where individual alignment is directionally agnostic at scale, cultural alignment guides users down a single path.

Whatever the model produces already looks familiar, so users accept it without noticing the nudge. Dissenting views receive less airtime, novel ideas sound eccentric, and taboo-breaking arguments never get to surface. Over time the model helps pin culture in place by delegitimising anything outside the frame. This makes cultural alignment a risky middle ground almost as persuasive as personalisation without the scrutiny that comes with universal alignment.

Stasis

One tricky problem with the cultural alignment project is that it claims to reflect what a society already believes, but in doing so risks arresting the processes by which beliefs change. Unlike universal alignment that seeks to drive us towards certain fixed ideas, cultural alignment looks at how we behaved in the past and updates the models accordingly.

The rub is that change lives at the margin. You don’t have to believe society is getting better to believe that the ability to shift your stance is worth protecting. Cultural alignment threatens that by mistaking the average for the ideal, and the present for the permanent.

Take same-sex marriage. In 1950, the dominant view in most Western countries was that it was wrong. A culturally aligned model, trained on that consensus, would have affirmed that position. You can patch the model, but it takes time to figure out that something has changed and push out an update.

Someone has to decide when opinion has moved enough to gather new data or commission fresh surveys. Then they need to re-deploy, audit for regressions, and push the update out across all downstream products. That will happen with all the speed of government bureaucracy, which means that the updates could trail real-world change by years (especially when the new view is still contested).

Relativism

There are basically two ways of thinking about the diversity of human values: value pluralism and value relativism.

Value pluralism holds that there are multiple, sometimes incompatible, goods that people can reasonably pursue (e.g. freedom, equality, or security) and that these values can’t always be reduced to a single master principle. It suggests that conflict between genuine moral values is tragic but real, and that choosing between them sometimes involves real loss.

On the other hand, value relativism claims that there is no objective way to evaluate values and that right or wrong are just whatever a given culture says they are. In its strong form, relativism rejects the possibility of cross-cultural moral critique. If a society condones slavery or subjugates women, that’s just their way of doing things.

The danger is that cultural alignment often confuses these two. It starts from a healthy respect for pluralism but slides into a kind of operational relativism, where any local norm becomes automatically valid simply because it’s local.

I describe myself as a pluralist in that I hold some values to be incompatible with basic moral responsibility. Without certain universal moral values, a maximalist programme of cultural alignment may endorse practices that many would see as troublesome. Not all the time, of course, but often enough to matter. Especially in places where dissent is already fragile and moral change depends on the courage of a few to challenge the many.

What type of AI do we want?

Cultural alignment is neither fine-grained enough to honour individual diversity, nor principled enough to serve as a moral foundation. It treats cultural averages as moral ideals, sidelining anyone who deviates from the script despite its best intentions. A better bet is to accept that people know themselves best and that some things are wrong no matter where you are.

That’s why my preferred approach looks like (a) a universal floor that guards against clear manifestations of bad behaviour, paired with (b) deep personalisation that gives everyone a model that acts in accordance with their values. You could layer culture on top, but only if it’s possible for individuals to override it in service of their own preferences.

Instead, we embrace the belief that different people can value different things within these boundaries in a way that is valid but uncomfortable. These tensions can’t always be neatly resolved, so while we ought to respect the clash we should also protect people’s ability to navigate it on their own terms.

I’m not saying personalisation is a silver bullet. A system that’s too eager to please may give us the moral world we already want, rather than the one we might strive for. Personalisation without restriction also risks people infringing on the affairs of others. And if my model is aligned to my values, and yours to yours, then what happens when we must coordinate?

This is partly why I’d prefer an approach that starts with a combination of universal values and stated preferences about the kind of person one wants to be. We give the model a principled foundation for its behaviour, rooted in our own moral identity to stop us from indulging our first order preferences (I want a cigarette) over our second order preferences (man, I wish I could stop smoking). Over time, small changes based on revealed preferences could refine this picture — but they should generally be subordinate to the user’s declared commitments.

A settlement on these terms grants that each of us is trying to live a life, and recognises that this effort is personal and plural. It’s not without pitfalls, but it builds from the right premise: that human beings are moral agents who deserve the right to choose.

Subscribe now

It’s 1894 in the city of La Coruña on Spain’s northern coast and José Ruiz y Blasco is painting a pigeon. He steps away from his easel for a moment, but when he comes back his thirteen-year-old son has finished the job.

The boy is good. Very good. So good that his father stops dead in his tracks. Ruiz had been painting birds for years. He was a trained artist, a respected professor at the city’s School of Fine Arts. He knew technique when he saw it, and resolved to give up painting when he thought his son had surpassed his old man.

The boy is Pablo Picasso, the great Spanish artist who liked to remind us that ‘art is a lie that makes us realise truth.’

The anecdote isn’t completely true — there are later paintings attributed to Ruiz — but the episode still gets remembered as a particularly resonant origin story in the art history canon.

Picasso built his reputation on rejecting naturalistic representation. He pushed Cubism into the cultural imagination because he knew that literal depictions of reality were a fool’s game. Instead, the Spaniard spent the best bits of his career celebrating how futile it is to show things as they appear to be.

And yet the story we tell about his youth is one of technical mastery. It’s about a bird rendered so perfectly that it convinced a professional painter to put down his brushes for good. Maybe the myth is necessary. Perhaps we need to believe Picasso could portray the perfect pigeon before we accept that perfect pigeons aren't worth the paint.

Then again, the young Picasso’s work wasn't actually flawless. It couldn't be because reality is infinitely complex and infinitely temporal. Any attempt to represent it is by definition an act of reduction. All paintings, no matter how detailed, exist in the margin between the thing and someone's attempt to show it to us.

In this sense, art is error.

For those of us interested in the AI project, it’s an idea that explains why some generative art looks kitsch or banal. When Midjourney produces images indistinguishable from a National Geographic photography exhibition, we get technical proficiency that rings hollow.

Yes, as models become more sophisticated they get better at representing reality. But that doesn’t get anyone’s blood pumping because the gap between what is and what we see can never be fully closed. It’s for this reason that the wise artist embraces the space between, rather than pretending it doesn’t exist.

Subscribe now

Beautiful errors

A few years ago I went to the Vatican. It’s a good day out, so long as you have nothing against metal detectors or being herded from one room to the next like cattle. Michelangelo’s ceiling of the Sistine Chapel is especially wonderful, mainly because it’s big enough (and far away enough) to be seen from deep within the crowds.

If you wriggle past the tour groups you might steal a glimpse at Raphael’s School of Athens, that monument to Renaissance idealism where each figure exists in exquisite harmony. Every line of perspective is satisfyingly calibrated and every face an expression of classical beauty.

The School of Athens depicts neatly proportioned bodies whose classical beauty is itself a kind of fiction. We still look at it because Raphael’s ‘perfection’ is a meaningful departure from reality, one that shows us a vision of human potential and intellectual harmony.

Alas, even error can become stale. What felt revolutionary in one generation becomes formulaic in the next. By the 19th century, Raphael's particular way of getting things wrong had been copied and systematised into predictable beauty.

What began as revolutionary techniques for representing reality became formulas for perfecting it. High art grew technically proficient but emotionally flat. Students copied masters who copied other masters, eventually creating a hall of mirrors that reflected the same forms.

Art needed new kinds of meaningful mistakes.

Every creative movement that mattered was a rebellion against the world that came before it. The Impressionists abandoned linear perspective for fleeting light. The Expressionists distorted faces to show emotion. The Dadaists threw coherence out the window by embracing absurdity as their organising principle.

Each of these groups recognised that art mediates between intention and execution, between what we can see and what we can depict. Close that gap too fully and you have something closer to documentation than art.

Every representation departs from reality in some way, but there are two very different approaches to dealing with that withdrawal (discounting attempts at accurate depiction). You can try to fix reality's messiness by smoothing away its imperfections in pursuit of beauty. Or you can amplify the world’s strangeness, pushing departures further until they reveal something that literal representation cannot.

Classical art tries to hide artifice by making its idealisations feel natural and its corrections inevitable. Revolutionary art celebrates deficiency, encouraging the viewer to reckon with the distance between reality and representation.

Picasso did the latter. In 1907, he produced Les Demoiselles d'Avignon, a painting that critics initially dismissed it as the work of a madman. Faces twisted into geometric fragments. Bodies viewed from multiple angles simultaneously. Colours that clash and lines that unsettle.

What Picasso saw — and what would define modern art for the next century — was that error could be the point. The mistakes that classical artists spent lifetimes learning to avoid could be reimagined as new ways of seeing.

The Surrealists had figured this out too, though they approached it from the opposite direction. Salvador Dalí’s paranoiac-critical method involved deliberately inducing a state of delusional perception, staring at random objects until he tapped into the ‘symbolic language’ of the subconscious mind.

René Magritte’s The Treachery of Images, which famously shows a pipe with the caption ‘this is not a pipe’, forced viewers to confront the fiction of representation. The idea is that these words make us conscious of the membrane between rendition and reality.

But it was the Cubists who gave us the most systematic approach. The likes of Georges Braque and Juan Gris formalised a new visual grammar based on fragmentation. Unlike previous movements that rebelled against specific techniques, Cubism rejected the fundamental premise of Western art since the Renaissance: that painting should create the illusion of looking through a window at reality.

Cubism was about information density. By deliberately breaking perspective, its painters could pack more knowledge into a single image than a camera could capture. They thought that art should convey our accumulated knowledge of something, not just a single encounter with it. This is partly what Picasso was getting at when he said ‘I paint objects as I think them, not as I see them’.

The golden age of broken AI

I bought this painting in 2019. It looks like a classical allegory, but with a flatness that reminds us we’re looking at the work of a machine. The faces are clearly human, but impossibly so.

It was produced by a generative adversarial network, a system in which two neural networks are locked in competition. One network (the generator) tries to create images convincing enough to fool its opponent, while the other (the discriminator) evaluates and scores the generator's output, pushing both to improve through competition. You can think of them as artist and critic.

GANs can be used for generating synthetic training data and detecting deepfakes, but they're known by most people as the class of models that brought AI art into the public imagination (there was of course Deep Dream, but it was a little more contained to the extremely online amongst us).

Early generative adversarial networks couldn't paint a convincing human face, but they could create portraits that existed in the uncanny valley between recognition and abstraction.

A GAN that learned to associate ‘face’ with certain patterns would dutifully reproduce them, even as the result looked like an acid trip. The machines were trying their best to paint like humans and failing, but the result was often better than they have any right to be.

We crossed a threshold somewhere between the flowing figures of early GANs and the hyperreal perfection of modern diffusion models (the things that make ChatGPT’s image generation function tick). The machines learned to stop making interesting mistakes by default. They became too competent and too reliable.

Early GAN outputs were obviously artificial, but they were more than synthetic slop. Compare that to your average image model today. In producing work that is too polished, they trigger a kind of emotional flatlining where we recognise a solid simulation and respond with indifference.

The labs have built systems that can render skin texture with exactness and generate lighting that obeys the laws of physics. Composing images according to the classical principles of beauty is for them light work.

But good art rarely tends to be so neat. Early image models were interesting because the machines had a thing for category errors. They saw features in noise, mixed up spatial relationships, and whipped up impossible architectures that felt emotionally satisfying.

Picasso spent his career learning classical techniques so he could break them. The best AI artists do the same thing by understanding how these systems function well enough to make them work poorly in the right ways.

Even in widely available image generators the tools are already there. Sliders that control how closely the model sticks to the prompt. Seed values that determine which accidents occur. Negative prompts that can force models to avoid trained behaviours.

Every AI system contains the germ of its own rebellion, if we're clever enough to cultivate it.

The most honest AI art puts artificiality to work. Images that look machine-generated but use aesthetic distance to reveal truths about its subject, or texts that sound alien but illuminate aspects of language we take for granted.

Glitch artists have been doing something like this for a long time. What they did with early digital systems, we can do with transformers, diffusion models or generative adversarial networks. The key is carefully orchestrated failure that reveals patterns invisible to conventional seeing.

I’m glad the systems got better, but I’m disappointed that the default way of using them emphasises their ability to accurately model reality. It should go without saying that there are plenty of artists who already use AI thoughtfully in their work, but the point is that the average user is drawn to faultless representation rather than beautiful error.

Instead, we ought to remember that there is no such thing as a perfect picture. Not for the ancients, not for the modernists, and not for us. Better to recognise that art lives between reality and perspective, and that the gap is where the good stuff happens.

Subscribe now

Last week I wrote about reasoning models. I argued that — despite some recent flawed work on the subject — they have some curious limitations, and outlined a rough sense of where I expect developers to go in the future based on those shortcomings.

While I was researching the piece, I read lots of recent critical writing about AI. Some of it was good, but much of it read like extremely wishful thinking about what exactly systems can and cannot do.

On the plus side, the experience did help me formulate a simple heuristic to sift through writing about our subject. As soon as I see someone call a large language model a ‘bullshit generator’, I know to take whatever follows with a grain of salt.

Usually that person is an academic. It should go without saying that not all critics are academics and not all academics are critics. But it seems to be critical academics whose voices carry disproportionate weight in shaping public discourse. On a personal level, they're a group I see more regularly as an academic researcher since leaving industry.

The type of person I’m describing is occasionally a technical researcher, but more often they are a humanities scholar. Normally it’s a person whose work I respect, an otherwise clever thinker who seems to have caught the bug. It’s an unfortunate state of affairs for someone who counts themselves amongst their number.

‘Bullshit generator’ is a kind of shorthand, one that many academics use to signal to others that they have the right opinions about the AI project. One person says it and then another. And just like that it becomes orthodoxy. Everyone you know rolls it out whenever the opportunity arises, so why shouldn’t you?

Our meme is recycled so consistently because it feels just naughty enough. You can put the phrase in a paper or a newspaper headline and no one will tell you off. It has a forbidden fruit quality to it. Can you believe what we just said!

The sociology of the thing is curious, but it doesn’t tell us why the idea itself — that large models are useless paper tigers that don’t ‘know’ anything — is so attractive in the first place.

I suspect it’s because many of them dislike AI, so they don’t follow it closely. They don’t follow it closely so they still think that the criticisms of 2023 hold water. They don’t. And that’s regrettable because academics have important contributions to make.

Subscribe now

Play the classics

I recently suggested that one reason for the animosity towards AI is that people feel like they’ve been duped. They had a vision of what AI ought to be in their head that doesn’t correspond to the technology in reality.

But strange as I think LLMs are, they are still useful. We’re talking about things that millions of people use every single day. Companies are openly saying job displacement is coming. Former US presidents agree.

But you wouldn’t think that was the case if you asked the average academic. They tend to scoff at the idea that anyone might use them for anything. You often hear them say things like:

• ‘It’s just linear algebra’

• ‘LLMs don’t know anything’

• ‘It’s all a PR exercise’

• ‘Stochastic parrot, stochastic parrot!’

• ‘Don’t they hallucinate everything?’

The most forceful of these is one is that trotted out reflexively: hallucinations. It’s all just made up, isn’t it? Don’t the models get most basic facts wrong?

Well no, not really. Certainly no more made up than some academic papers. You might have had a case back in 2023, but these days hallucinations are much rarer than you think.

On the Hugging Face hallucination leaderboard, the top four models score a factual accuracy rate of more than 99% on a document summarisation benchmark.

You might say that the test isn’t fair game because LLMs do more than summarise. And you would also be right to point out that some of OpenAI’s newer reasoning models seem to have bucked the trend based on the SimpleQA and PersonQA benchmarks.

But the rest of the stats tell a different story. On the Simple QA leaderboard, the best performing models — those that tend to supplement answers with an internet search functionality — clock in with between 90 and 95 per cent accuracy.

Fine. Even if they can regularly produce factually accurate information, they still don’t really know anything.

The problem with this line of thinking is that it requires a bit of philosophical wrangling, one that (for reasons unclear) the vast majority of academics seem unwilling to engage in. This is particularly frustrating because if you’re going to make forceful claims about epistemology, it seems rather unsporting to dodge the resulting debate.

When you think about these questions for more than five minutes, it’s pretty obvious that terms like ‘knowing’ or ‘understanding’ are slippery concepts. Never mind ‘truth’ or ‘information’. I don’t feel confident to say much other than AI definitely knows something.

Occasionally the claim gets tighter and it becomes something like LLMs can't generalise from a small amount of data, but performance on the ARC-AGI benchmark with just a handful of examples seems to prove that isn’t actually the case.

We also have thinking or reasoning. This one basically says the machines don’t think because that’s only something that humans can do. At best, all they can do is simulate thinking. This one I don’t mind so much as at least it tries to engage in a substantial argument that gets at the core of the thing.

It might be the language models can only simulate thinking or reasoning. Call me a utilitarian, but what matters to me most is how effective they are in the real world. Whether or not they are simulating thinking has no bearing on whether or not the machines are capable of rearranging the world for better or worse (though if you want to read about what I think is actually happening inside LLMs you can do that here).

And of course there’s the Foucaldian take: it’s all a PR exercise. Obviously, companies like to promote their product. AI is no different in that respect. But to argue that the richest firms in history are deploying trillions of dollars of capital in service of PR is a total non-starter.

You could say that they should be more sceptical of their inventions, but to propose that the entire apparatus of AI development — fighting off competition for chips, building enormous datasets for pretraining, and fine-tuning the model with the help of thousands of human reviewers — is for reputational purposes strikes me as a bit far-fetched.

How to criticise AI

To be clear, I am not down on academics. I am one! I only wish my colleagues would think more critically about their own beliefs, and accept that we simply don’t have enough information to understand where the ceiling is for the AI project as it exists today.

Below are some suggestions for what better AI criticism looks like (inspired by this excellent post) that reflects this uncertainty. It’s not exhaustive, but it gives a rough survey of useful elements for formulating critical commentary.

Things to do

• Stay current: Base your claims on recent capabilities by staying up to date with AI research, model deployments, and real-world usage. When critiquing, use the best available models — not convenient strawmen (thankfully we are past the era of slide decks filled with GPT 3.5 gotchas).

• Embrace humility: Accept uncertainty as a starting point and modify your approach accordingly. No one fully understands these systems yet (including the people building them). All things being equal, curiosity should precede criticism. In the words of Erling Haaland, stay humble!

• Study adoption: Some struggle to believe anyone is actually using AI. But they are. Millions of them. If you want to analyse failure modes, you’ll have plenty go at by talking to the doctors, lawyers, and students who use the models. But you’ll also see that not every use-case is malicious (and that people are actually using LLMs).

• Sample widely: When models work, seek to understand why and under what conditions. When they fail, collect multiple instances across different contexts. Ask the same question. A single amusing error tells us little; patterns of failure (and success) across varied conditions reveal the actual boundaries of capabilities.

• Be creative: If LLMs don't fit neatly into existing epistemologies, maybe it’s time to make new ones. Rather than forcing these systems into old categories or dismissing them for not fitting, have some fun by developing new conceptual tools. Create the language and frameworks we need to understand AI.

Things to avoid

• Reductive claims: Related to the above, saying ‘it’s just pattern matching’ explains nothing on its own. If you must make reductive claims, embed them in substantive arguments about what follows from that reduction. Ask whether your reduction captures what matters. Then explain why.

• Forecasting with confidence: The history of AI is littered with assured proclamations about what machines will ‘never’ do. Current limitations are empirical facts worth documenting, but extrapolating them into fundamental barriers rarely ends well.

• Treating AI as a monolith: Remind yourself that different architectures, training methods, and deployments yield vastly different capabilities. And note that systems are often composites. Understanding which component does what is crucial for meaningful critique.

• Cherry-picking: Only citing failures while ignoring successes or dismissing benchmarks that contradict your thesis sounds more like advocacy than scholarship. Intellectual honesty means engaging with the full empirical record, especially the parts that surprise you.

• Credentialism: Yes, peer review still matters. But dismissing research because it comes from industry labs or preprint servers rather than traditional journals is self-defeating. In a fast-moving field, the most important findings often emerge outside conventional channels.

Uncharted waters

Many moments in the history of thinking machines can be described by the maxim fake it until you make it. Too often what looked to be impressive performance was contingent on the man behind the curtain. That’s a thread that runs right through the invention of the difference engine to the emergence of parallel distributed processing in the 1980s.

But that isn’t happening today. Yes, today’s large models are complexes of data, human input, hardware, and clever algorithms. But they do actually work well for the most part, which is why millions of people use them every single day. In that sense our moment is unprecedented in the history of AI.

But right now, many academics who speak to policymakers or the press badly underestimate the capabilities of the best models. They dismiss LLMs out of hand and don’t engage with the substance of the technology.

Media narratives skew sensational or simplistic, and policymakers end up getting the wrong end of the stick. This is clearly bad if you want to make sure AI is integrated into society in the most socially beneficial way possible.

Accepting the reality of the situation is the best way to produce timely and relevant work. But that requires getting familiar with the technology so that the public debate is grounded in clarity.

AI is a social, cultural, and philosophical event. These are qualities that should make the technology the business of academics. Some are already doing great work, but more are needed to ask the questions the engineers don’t. What do humans do in a world with advanced AI? What kinds of collective failure modes exist when we all begin to use LLMs? And how should these systems be trained, evaluated, governed?

These are human problems, but too many scholars have absented themselves from the conversation. They think refusing to engage is a form of critique, when in fact it’s a form of abdication.

If they wanted to, academics could help define the terms of safe development. They could map the new epistemologies these systems generate, trace their impacts, and build the intellectual scaffolding we need to live alongside them.

But for that to happen they need to accept the AI project for what it is, not what they wish it to be.

Subscribe now

Reasoning models are funny things. You make one by taking a vanilla large language model, asking it to produce a chain of outputs on its way to some final goal, and encouraging it to follow these steps in sequence before generating an answer.

They work like a charm, mostly, and are behind some of the more impressive examples of AI applications. Reasoners are sitting pretty at the top of lots of well-heeled benchmarks, and have even caused some critics to think again about the limits of the current paradigm.

Still, not everyone buys it. The more sceptical amongst AI watchers like to argue that what looks like thinking is just a trick of the eye. It’s a mirage, or as Apple put it in a paper last week, an illusion.

To make their case, researchers used puzzles to test how large reasoning models handle increasing problem complexity. They took puzzles like the Tower of Hanoi, converted them into textual descriptions, and fed them into some of the best models.

The results didn’t reflect well on reasoning models, showing that — when faced with a sufficiently high level of complexity — they pass a tipping point beyond which performance collapses. Despite having more compute to play with, the research shows that models tend to throw in the towel when they deem the problem to be sufficiently thorny.

Lots of people can’t help but think that this time large language models really are in trouble. Doesn’t this prove they aren’t actually thinking? Are reasoning models useless? Shall we cancel the short timelines?

In short, no. This is a persuasive but flawed bit of research, one that let a desire to say something provocative get in the way of what might have been solid work. One obvious problem is that they authors don’t even define ‘reasoning’ or ‘thinking’ in the paper. It seems odd to call what LLMs are doing an illusion if you don’t bother to explain what they are pretending to do.

Likewise, some of the ‘high complexity’ problems require more reasoning than fits in the context length (writing out a reasoning trace for Tower of Hanoi with 20 disks would take months for a human). This is probably why the models call it a day when the researchers’ prompts collide with RLHF’d objectives like ‘be concise’.

Finally, and this is the big one, they don’t let the models use tools. The test is about what we might call ‘pure reasoners’ without access to the simple system elements that would make these puzzles trivial for any consumer-grade LLM.

Subscribe now

Scale models

We’ve been here before. First large language models were useless toys. Then they got bigger, and boy did they get better. Eventually they could handle textual and visual inputs. More recently, the advent of reasoning models dislodged some of the tougher benchmarks like ARC-AGI 1.

The same pattern repeats itself. Some experiments seem to erect an insurmountable barrier for the large model paradigm. Then a new factor like scale, multimodality or reasoning helps models blast through the wall. Sceptics keep calling the end of the line, but the train doesn’t seem to notice.

They forget AI is a moving target. Today’s models are vastly more complex than those of just a few years ago. Sure, they are bigger and they use chain-of-thought techniques, but they also contain specialised modules that allow for tool use, memory, sand-boxed computation, and internet search.

The models are kind of sticky, which is probably their most underrated characteristic. You can build on top of them, giving them new capabilities that help overcome what used to seem like irreparable flaws.

Apple’s test stumped them because it only dealt with the raw model without its supporting infrastructure. With no tools, search or visual processing, they were playing blindfolded with one hand tied behind their back.

The mythos of large language models is all about scale. The jump from models like GPT-2 (1.5 billion parameters in 2019) to GPT-3 (175 billion parameters in 2020) famously showed us that making models bigger and training on more data led to remarkable gains in generalisation without task-specific training.

Scaling brought better fluency, coherence, and coverage of knowledge. Large models began to work well on tasks they hadn’t explicitly seen before, probably because they had absorbed a head-spinning number of patterns through their pre-training process.

Yes there were some scale maximalists, but by the mid 2020s lots of researchers generally accepted that simply making models bigger wasn’t going to cut it. The performance curves for some challenges were flattening despite the growing size of models.

Even until last year, it wasn’t clear that LLMs would be capable of clearing certain evals designed to test for general reasoning capabilities. The most famous of these tests was the Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) created by deep learning researcher François Chollet.

The test was originally launched in 2019, but the top rated scores hadn’t improved much up to the launch of last year’s competition. That was because Chollet designed the test to include novel problems, which means—even ingesting massive chunks of the internet—large models were unlikely to have seen a critical mass of similar examples in their training process.

According to Chollet, because language models only apply existing templates to solve problems, they get stuck on tests that human children would be able to manage comfortably. But the test also stood for so long because it wasn’t particularly well known, which narrowed the pool of researchers trying to solve it. Then came the prize launch in the summer of 2024, which offered cash and kudos for passing the test.

When the test was launched, I suggested that a model would pass it in fairly short order:

“The (quite literally) million dollar question is whether ARC-AGI will stand the test of time. If I had to guess, I would expect major progress on the challenge within the next year or so. This is because a) bigger models with some clever algorithmic improvements seem to be doing something other than simple pattern matching; and b) there’s already been some improvement on the benchmark since it was released.”

I didn’t have access to any kind of special information when I made that prediction. I just thought that models were already much better than they had any right to be, and that it was unwise to bet against systems that can leverage more compute (which turned out to happen at inference time via reasoning models).

A few early entrants improved baseline scores to more respectable levels, but it wasn’t until OpenAI’s o3 model scored 87.5 percent on the ARC-AGI benchmark (albeit via a very costly process) at the end of last year that we could say it was passed.

Chollet himself, a well known critic of large language models’ ability to reason, said ‘all intuition about AI capabilities will need to get updated,’ while Melanie Mitchell (also a long-time sceptic) called it ‘quite amazing’.

With models performing well on the original test, Chollet set about on a follow-up benchmark called ARC-AGI 2. Released in May earlier this year, ARC-AGI 2 ‘raises the bar for difficulty for AI while maintaining the same relative ease for humans.’ Currently, the best performing model is a reasoning version of Anthropic’s Claude 4 Opus at 8.6%.

Apples and oranges

In Apple’s paper about illusory thinking, their analysis of the reasoning traces (the step-by-step thoughts the model generates) show that the model’s ‘thinking’ is prone to lots of different failure models.

They bundled behaviour into three regimes of problem complexity, which each saw the models perform with varying degrees of success:

• Easy problems: As expected, the models performed best on these — in several cases achieving solid scores. When they did fail, it often involved finding a correct solution in the chain-of-thought before exploring wrong alternatives. This ‘overthinking’ has been observed elsewhere, and leads reasoners to sometimes talk themselves out of the correct answer by the end of the chain.

• Moderate problems: Again, the models could succeed here. Failures, when they did happen, tended to involve generating a slew of incorrect intermediate steps from the get-go. When they worked, only later in a convoluted reasoning chain did they find a correct solution (not exactly efficient but in general this seems fine to me).

• Hard problems: This is the ‘collapse’ scenario described by Apple. The model’s chain-of-thought isn’t pretty to look at, consisting of numerous steps that seem to be incorrect or irrelevant. There is no point in the chain where it finds a workable approach (though as above this is likely because some of the correct answers exceed token limits).

One curious example the researchers describe is the Tower of Hanoi puzzle (a classic problem that requires moving disks between pegs under strict rules). Apple’s team tested their LLMs on Tower of Hanoi puzzles of increasing disk numbers, finding — perhaps unsurprisingly — that performance fell as the number of disks grew.

Then they tried an ‘algorithm injection’ experiment where they gave the model the correct algorithm in the prompt (essentially walking it through the steps it should take), to see if that helped on harder cases.

The result? It didn’t help at all. Even when told exactly how to solve the puzzle, the reasoning model could not execute the steps reliably once the problem became sufficiently complex. The group doesn’t really explain what they think is happening here, but they do suggest it represents ‘limitations in performing exact computation’.

This is no doubt true, but large language models have always been pretty bad at performing exact computation. That’s why people ask ChatGPT to ‘use code’ when making calculations if they want to get a reliable answer. Had the model been allowed to use the algorithm via some plug-in, it would have been a different story.

Differential reasoning

Ok, despite the test not being a total wash, there are some limitations to reasoning models. Why is that?

I’ll begin by saying no-one really knows for sure. Not the researchers in the AI labs, not the markets, not academics, and certainly not me. All we have are intuitions. Mine is that it involves the symbol grounding problem, which concerns how symbols like words can acquire intrinsic meaning.

Reasoning systems, like all LLMs, find correlations and produce patterns. But as Apple’s work reminds us, they can also produce explanations that don’t necessarily correspond to the basic facts of reality. They contain oodles of knowledge, but it’s more like raw ore than refined metal.

When reasoners go doolally, it’s because they struggle to reliably connect concepts to an underlying model of the world. Reliably is the key word here. I do think the models can ‘reason’, providing your definition is something like ‘the systematic chaining of relationships between internal representations to reach a conclusion that satisfies a given set of constraints.’

This rather broad definition accounts for reasoning by navigating the web of learned similarities among representations, where an agent steps through them until a pattern that satisfies the goal appears. That is how I think an LLM reasons, which I do think is reasoning — but it’s not exactly the same as what people do.

Think about two basic ways of knowing about a tree. The concept ‘tree’ makes sense because it isn’t ‘bush’, ‘pole’ or ‘cloud’. Large models gobble up billions of sentences, notice the connections between tree and its neighbours (leaf, bark, shade, roots), and build a high-dimensional map where the concept’s position is fixed by everything it is not.

Ask a model ‘what climbs a tree?’ and the word squirrel lights up because it sits only a few degrees away in that semantic constellation. This is grounding by difference, where each representation is determined by the relative positions of other representations.

Purists would say this isn’t really ‘grounding’ at all because the model is only grappling with the meaning of symbols by using other symbols. Compare that to my own experience of walking up to a tree, touching the trunk, and feeling the cambium under my fingernail.

That multisensory encounter anchors the word tree to a slice of the physical world. A logger, a robin, and a child building a tree-house all ground the concept in lived affordances (in that you can chop it down, nest in it, or climb it). This is grounding through reference.

LLMs don’t mistake a tree for a toaster because their vector space keeps those poles far apart under default conditions. Instead, they may hallucinate a ‘glass roof’ that provides ‘ample shade,’ because no tactile or optical reality is acting as a check on these associations. Humans catch it instantly because reference knowledge (e.g. glass is transparent, shade needs opacity) is wired into our practical intuition about the world.

Based on these ideas, we can try to make sense of the failure modes described by Apple:

• In easy tasks, the tree of possibilities is more likely to contain useful examples the model has seen before. The model grabs the answer early, then keeps sampling neighbours until it drifts off course. Once the local similarity signal weakens, nothing tells the sampler it has overshot the target.

• In moderate tasks, the answer lies a few hops away. The model finds its way through a cloud of wrong patterns until it lands on a cluster that lines up with the goal state. More tokens buy it more opportunities to search the manifold of token differences.

• In hard tasks, the model doesn’t cover itself in glory. Beyond a certain depth there is no nearby cluster that satisfies all constraints, which means the sampling process has nothing to feed on. At harder puzzles the model stops thinking either because the next steps all look equally bad (or because it can’t fit the answer in its output due to token limits and RLHF constraints).

Grounding provides the foundation for reasoning, but it isn’t the same thing as reasoning itself. Rather, you need grounding to tether reasoning to the world it claims to explain. So when models ground using difference, I like to think about this process as a kind of differential reasoning.

This is why models work so well in some instances and fail badly in others. The frontier is jagged because they reason in a different way to people.

Beyond pure reasoners

I don’t see this observation as a major bottleneck for AI development. In fact, I think two possible responses put developers in a remarkably strong position: systematisation and agency.

Systemisation is about making the core model a node within a bigger apparatus. We keep the language model in place, but surround it with specialist gadgets. Web search look-up, a code sandbox, a vision encoder, and a knowledge base. The model doesn’t need to have all the answers, it just needs to decide when and how to invoke the right tool.

In practice this is already how people tame hallucinations. On the Simple QA leaderboard, the best performing models — those that tend to supplement answers with an internet search functionality — clock in with between 90 and 95 per cent accuracy.

Each add-on is there for a reason. External search keeps the model up to date, an execution engine allows it to deal with hard arithmetic or code, multimodal functionality lets it ground words in pixels, and long-term memory means it can recall prior interactions.

Apple’s experiment stripped all that away. Re-running their Tower-of-Hanoi test with a tool-using agent would let the language core sketch a plan, hand the plan to a symbolic solver, and verify the result before answering.

Systematisation aside, a second approach might even allow models to ground via reference. Instead of static prompts, drop the model into an environment where it can act, observe consequences, update its policy, and store new skills. This is the play in David Silver and Rich Sutton’s ‘Era of Experience’ paper, where the reward signals come from the environment rather than a human ratifier guessing from the sidelines:

‘Such agents will be able to actively explore the world, adapt to changing environments, and discover strategies that might never occur to a human. These richer interactions will provide a means to autonomously understand and control the digital world.’

In the short run, bolting on tools will keep pushing the envelope of what today’s text-trained models can accomplish. But in the long run, the core of the models still lack first-hand reality checks. Grounded experience offers a more durable solution, but only if we close the loop so that actions have consequences the agent can’t ignore.

The great AI researcher Marvin Minsky argued that the human brain was what computer scientists call a ‘kludge’. He thought that grey matter was an inelegant solution to the challenges faced by early humans, cobbled together from specialised parts over the course of millennia.

I like the kludge concept because it suggests intelligence is a product of both specific mechanisms and their patterns of interaction. For AI, the implication is that lots of small, dedicated modules can be linked together to form a system that benefits from their associations.

It’s a useful idea for seeing what the future looks like. Yes, today’s pure reasoners have limits. That’s why we ensconce them in systems that prevent those frailties from manifesting far more often than they might otherwise do. But really that’s just a stop gap, a way to get extremely capable models that can perform a kind of referential reasoning via the back door.

Sooner than you might think, the labs will produce a tool-using LLM that works in the wild and gets better based on what it sees. When that happens — and it will happen — today’s pure reasoners will look like toy models.

Subscribe now

Back in February I wrote about slop. I argued that what we call ‘slop’ is the unfortunate byproduct of widening the means of creative production. It’s a kind of runoff that pools wherever friction is low, one that accumulates when the marginal cost of producing a creative artefact converges on zero.

Since then I’ve been wondering about what slop means for visual culture. An obvious reading flows from Herb Simon’s adage that an abundance of information leads to a scarcity of attention. Slop in this sense is a pollutant that stops us from wading through the digital commons.

This is surely true, but this kind of analysis risks getting the wrong end of the stick.

At its core, the age of AI content is one where more people are able to produce creative things. Some will be good and some will be bad, but in absolute terms there will be more great art than ever.

How could there not be? We’re talking about tools that give kids in their bedrooms the same kind of firepower as a small production studio. People thought that moment was a couple of years away, but recent developments suggest shorter timelines might be more accurate.

The problem is that you have to claw your way through the sludge to get to the good stuff. All things being equal, I personally do not want to see more inane, self-referential, and stupid stuff on the internet.

Of course, all things aren’t equal. Slop is less target than collateral damage. It’s the thing that we put up with because doing so means that anyone can make stuff that was once prohibitively expensive.

The future I see is a sloppier one, but it’s also one with more good art than there has ever been before. More thoughtful films. More provocative performances. More high quality writing.

But the slop-for-creative-abundance bargain only works if you can sort the great from the good. For the deluge to be worth the price, you need to be able to reliably choose the stuff that best aligns with the type of person you want to be — and the type of person you’d like others to see you as.

You need to have good taste.

We’re going to have to paddle a little

Each new creative tool — moveable type, cheap lithography, desktop publishing or blogging platforms — has followed the same pattern. A glut of new work. Howls about falling standards. And a recalibration of what counts as good.

The lesson, if such a thing can be drawn from a study of slop across the ages, is that art gets cheaper to make over time. That lowers barriers to entry, which in turn allows more people to produce more art.

Even today, before there was ‘AI slop’ there was ‘Netflix slop’. Slop the elder can be tricky to describe, but it’s basically something like technically sound narrative emulsified for mass consumption.

Netflix is a useful thing in that it reminds us that a surplus of content creates new social worlds, but it doesn’t completely do away with the shared cultural artefacts that demarcate the ‘mainstream’.

The point is that emerging cultural touchstones maintain the capacity for making and breaking tastes. We live under the illusion that personal preferences are already unmoored from the taste of others.

Except that’s not quite true.

Yes, your X feed is not the same as mine. Your Amazon Prime recommendations look very different. But we are using the same types of viewing devices in the same familiar settings. We watch in the same kind of way.

The screen is the monoculture. It forces a shared cadence (swipe, tap, binge, skip!) even when the titles differ. That alone represents a flattening of behaviour into something we might reasonably call the mainstream.

As for what we watch, it’s true that there are a thousand shows that will never cross your path. But there are many that do, those that become a talking point in the right circles (or the basis for public policy if you live in England). These moments are thinner and faster than Dallas cliff-hangers, but they still constitute a form of mass culture.

In the broadcast age, the bundle of shared reference points that let strangers triangulate a conversation were large and stable. Today, they still exist — in that at any given moment there is a thing that lots of people are likely to know about — but its constituent parts are small and fast-moving.

Content (not to be confused with art) may already be more readily available than ever, but it’s still funnelled towards the viewer by platforms that stand in for tastemakers. You may never voluntarily watch a MrBeast video, but you still know who he is.

Subgroups flourish inside this architecture, but they are still constrained by it. Each micro-culture gets its own fifteen minutes of fame because ‘mainstream’ and ‘niche’ exist as something like phases of attention. The greater the volume of content, the more frantic the jostling for the moving middle.

Video and audio generation models will drag this process to its inevitable conclusion. There is simply no way today’s cultural settlement remains stable in a world where it’s possible to create studio-quality stuff from your bedroom. Yes, we already have YouTube, but fan-made films aren’t exactly at the same level as your average Hollywood flick.

The excess of choice will mean we can no longer ride the current and assume the best bits will wash ashore. We’re going to have to paddle a little.

In my slop era

As the world gets sloppier it produces a desire for people to find the things that matter most to them. This is the impulse that is driving the emergence of the ‘tech-literature guy’, the person who is as familiar with Kate Chopin as they are with mechanistic interpretability.

People want to find steady points of reference amidst all the content. One way to do that is to revisit (or I suppose visit may be more accurate) the classics. You might begrudge the tech literature guy, but I generally think more reading is an unambiguously good thing.

Where it gets more interesting is as signal. What’s the point of reading one-thousand pages of Infinite Jest if you don’t tell anyone about it? That was important before the age of generative media, but slop has sharpened the point.

As the cost of making falls the cost of finding begins to rise. That premium breeds anxiety. Editors, commissioning boards, and gallery curators once filtered the sludge before it reached the public. Abundance overwhelms those dams and forces us to train an inner critic.

Think about it. Preference is basically a matching problem, and the profusion of AI tools creates the potential for art which is tailor-made for you specifically. That’s all well and good, but it comes with a catch: hyper-personalisation removes the last outside check on quality. Institutions, critics, and even your artier friends can’t pick the best possible option for you.

More creative artefacts will be produced. Slop will keep bubbling to the surface. The old referees will keep losing ground. But the faculty that tells you ‘this matters’ or ‘that doesn’t’ is immune to scale and indifferent to recommendation engines.

Taste can’t be automated, which is why its value appreciates in direct proportion to the noise that surrounds it. So if the age of slop feels destabilising, that’s just the weight of responsibility shifting to the individual.

I think the fixation with AI slop — and the fact that many commentators seem to completely misread what it is and why it exists — is born of anxiety. It’s anxiety about what happens in a world where the creative classes are no longer the cultural gatekeepers.

But it’s also anxiety about the need to be discerning. When there’s more to choose from then ever before, taste appreciates as cultural currency. If you’re worried about how to pick the bitter from the better, then you’re going to have a thing or two to say letting any old person make stuff and share it with the world.

Subscribe now

A few years ago, I was in a seminar about medieval alchemists.

The researcher, a historian of science, was interested in what those who practiced the arcane knew about the natural world. Probing to get something out of a quiet group, they asked a question I still think about: what is knowledge?

Things took a circular turn. Knowledge turned out be one of those things that resists precise definitions, a lumpy mixture of truth, belief, rationality, and instinct. One popular idea was that knowledge is processed information, and that information becomes knowledge when we understand it.

This sounds neat enough, but there are some problems here. You can of course internalise false information. You can believe true things for the wrong reasons. And you know how to recognise a face but not how you do it.

Our medieval alchemists knew things, didn’t they? They knew about mercury, transmutation, and the four elements. These were coherent ideas, but they were ultimately wrong.

‘Understanding’ is just as slippery. We might feel that we understand something because it fits our worldview, not necessarily because it corresponds to some fact of reality that we know to be true. The broader point is that knowledge and knowing are not so clear cut. Even information, which seems stable enough, is messy, contingent, and shaped by who’s looking and what they expect to find.

In this essay, I think through some of these problems and apply them to AI. I want to make sense of claims that even the best models don’t really ‘know’ anything, and the counterclaims that argue for the opposite position.

What follows are five different ways of knowing and what they mean for getting to grips with thinking machines:

1. Knowledge as representation

2. Knowledge as practice

3. Knowledge as situation

4. Knowledge as power

5. Knowledge as emergence

I argue that AI, by which I mean frontier models, partly satisfies the criteria associated with each of these ways of knowing. It doesn’t tick all the boxes, but it does enough that I suspect AI knows more than many people give it credit for.

Knowledge as representation

In the Western imagination, the oldest description of knowledge is as a representation of reality. Way back in the 4th century BCE, Plato and friends mulled over ways of knowing in Theaetetus.

One of the more compelling ideas put forward is that of knowledge as a combination of true beliefs about the world and a compelling reason for that belief.

But they don’t buy it, at least not in every circumstance. The dialogue rejects the idea on the grounds that you must have the right kind of justification. It needs to be one that belongs to the person making the claim, it needs to show why the belief is true, and it needs to explain the belief by getting at a root cause.

I am speedrunning, but today we call this type of knowledge the justified true belief (JTB) model. In JTB, we might say that knowing that the Earth orbits the Sun would mean (a) it’s true (the Earth really does orbit the Sun), (b) one believes it, and (c) one has justification (e.g. scientific evidence or sound reasoning) for that belief.

Knowledge in this view is a mental representation that corresponds to reality, one that is underwritten by a justification. It’s essentially a mental mirror of the world that is true and warranted.

The model’s heyday ended with a famous paper by the philosopher Edmund Gettier in 1963. Gettier devised clever scenarios (now known as Gettier cases) showing a person could have a belief that is true and well-justified, yet we would hesitate to call it knowledge because the truth resulted from luck or coincidence.

For instance, imagine you glance at a normally reliable clock which by chance stopped 12 hours ago. Your belief about the current time is true and justified by said clock, but only accidentally so. This is sort of what Plato was getting at: it is possible for a justification to become disconnected from the truth, which melts the causal link between the elements that make up real knowledge.

There are many critiques of the idea of knowledge as representation. Far too many to get into here. But one of them is worth mentioning: the idea that knowledge doesn’t really mirror the world.

John Dewey defined knowing as an active ‘organism–environment interaction’ rather than a static representation. Other pragmatist thinkers like Richard Rorty agreed. They might say that knowledge is better seen as a tool for coping with the world rather than a reflection of it.

What about AI?

Let’s first assume we’re dealing with an agent capable of computer use, one that I’ve asked to do my shopping. We have to jump through a couple of hoops here, but my view is that a sufficiently capable model knows some things in a JTB sense.

It carries a representation (‘Harry prefers Shop A’) that guides its belief in what I want it to buy. The representation is true in that it correctly knows what I want and it has a good reason for thinking that because it has a log of my previous choices.

JTB only asks for a belief-state that produces the right behaviour under the right evidence. If our agent carries a memory module that always makes it choose Shop A for me, that module plays the functional role of a belief system. (But only if we adopt what Dennett calls the intentional stance, where we treat a system as if it literally has beliefs, desires, and rationality whenever that helps you forecast what it will do.)

So, accepting some mental gymnastics a system can just about tick the JTB box. What it can’t do is satisfy Plato’s account, which has much harsher conditions for real knowledge. There are a few reasons for this, but the main one is that the agent needs an explanation (that is, logos) that reveals the cause or essence of the thing and shows why it cannot be otherwise.

Of course, we don’t tend to apply these conditions to all types of human knowledge. I haven’t actually conducted experiments to prove the Earth orbits the Sun, and my maths skills are a little too rusty to figure that out using first principles.

That’s why the JTB version of knowledge as representation became popular: because it stops us from throwing the baby out with the bathwater by saying humans know very little.

But the JTB recipe proved brittle. Gettier showed that beliefs can be true and well-justified, yet still only accidentally correct. In other words, an AI may pass the minimalist JTB test the same way we often do (provisionally and with a dollop of luck) but that’s not the same as self-explanatory knowledge.

Knowledge as practice

But is knowledge something we only hold in our head? We ride a bike, catch a ball, spot a friend’s face in a crowd. We do those things instinctively. The know-how resides in coordinated muscles and perceptual cues rather than in explicit propositions.

Because so much competent action relies on background noises that resist full articulation, treating knowledge as solely internal representations misdescribes everyday expertise. This idea is best captured by Michael Polanyi’s famous adage: ‘We can know more than we can tell.’

This view moves from knowing-that (viewing knowledge as factual or declarative) to knowing-how (the practical mastery that comes from doing and experiencing). Instead of a mirror held up to reality, knowledge is seen more like an ability we use in the world.

The philosopher Gilbert Ryle made a similar point in 1949. Ryle skewered the assumption that all knowing-how (like swimming) is really just a complex kind of knowing-that (knowing facts or rules about how to swim). After all, one could know how to swim even if one cannot articulate the physics of swimming.

In cognitive science, what we call the ‘phenomenological tradition’ emphasises the body as the locus of knowing the world. Maurice Merleau-Ponty thought that our body knows how to reach for a cup or navigate space in ways we don’t usually conceptualise abstractly. Perception, in other words, is itself an active engagement with the world.

Others took this idea further. They argued for an ‘enactive’ view where knowledge emerges through dynamic interaction of an organism with its surroundings, rather than through constructing internal representations detached from action.

The philosopher Jerry Fodor famously doubted these ideas. Focusing on Polanyi, he reckoned we should be careful calling something ‘knowledge’ if it can’t be verbalised or symbolised. Polanyi might respond that tacit knowledge underpins explicit knowledge. It supplements, rather than supplants, the process by which we make sense of explicit information.

What about AI?

In the 1960s, Hubert Dreyfus argued that early artificial intelligence efforts faltered because they misunderstood human knowledge as rule-bound rather than learned. He argued that a chess master ‘just spots’ the right move rather than crunching numbers, acts from a sort of embodied familiarity, and cannot fully articulate the know-how guiding the act.

Dreyfus was talking about the symbolic school of AI that uses hard-coded rules, but he was also sceptical of machine learning because it struggled to get at the underlying meaning.

Modern systems, he might say, process patterns and find correlations, but they produce explanations that don’t necessarily correspond with the basic facts of reality.

This is a version of the symbol grounding problem, which concerns how symbols like words can acquire intrinsic meaning (rather than being defined only in terms of other symbols). A model might wax lyrical about molecular biology, but we don’t know whether ‘molecular’ or ‘biology’ are stable concepts that refer to in-the-world properties.

The problem, of course, is that there’s no real way to determine whether or not today’s systems have some sort of sophisticated world model that emerged as a fortunate byproduct of next token prediction. Maybe they do and maybe they don’t. My personal view is that the proof is in the pudding. If AI starts to consistently discover new facts about the world without any hand-holding, then it’s probably we who misunderstand the nature of knowing.

For now, though, let’s be clear about what we’re saying as it relates to practised knowledge. There are basically three major claims here:

1. Skilled action emerges through repeated copying.

2. Know-how is stored in dispositions rather than explicit propositions.

3. Embodiment anchors meaning in the world.

As I see it, the most sophisticated models satisfy both the first and the second criteria. My shopping bot does improve by trial and error, and its knowledge does emerge from experience.

The last claim is where it falls short. The shopping agent still lacks a full sensorimotor experience of being that Dreyfus and the phenomenologists argue is necessary for the richest kind of human know-how.

Until an agent can live in an environment, develop a durable feel for what matters, and draw on that feeling in the real world, its tacit knowledge stays closer to habit than to embodied skill. Of course, it’s not all that clear to me that practised knowledge must also be embodied — but that’s a debate for another time.

Knowledge as situation

Donna Haraway coined the term ‘situated knowledges’ in a 1988 essay arguing against the illusion of pure objectivity in science. The idea is that all knowledge is partial, that the act of knowing is not only dynamic but also shaped by the relative position of the knower.

Here the knower is influenced by the historical, cultural, and physical conditions that shape their ability to form knowledge. It’s an idea closely related to Thomas Nagel’s ‘view from nowhere’ schtick, the philosophical ideal of stripping away local standpoints until you see the world independent of any particular observer.

Science often tries to reach this altitude. We describe colour as wavelength, love as neurochemistry, or the self as a biological organism. Each move feels like progress toward objectivity.

But Nagel argues the project is impossible to complete and perilous to over-extend. The very act of thinking is anchored in a ‘view from now-here’ that emanates from a physical entity with a history of being.

More concretely for our purposes, the historian of science Thomas Kuhn foregrounded the role of perspective with his concept of paradigms. In The Structure of Scientific Revolutions, Kuhn showed that scientists’ interpretation of data is conditioned by the ensemble of theories, methods, and exemplars they have at their disposal.

In Kuhn’s view, what scientists see as facts (and even what they observe through instruments) is filtered through the pores of conventional understanding. When paradigms are washed away by the storms of scientific revolution, scientists resurface to find a different world of knowledge.

Critics of these ideas charge philosophers with relativism.

They often ask: if all knowledge is perspective-bound, can we say any fragment of knowledge is better or more true than another? This was at the heart of the 1990s Science Wars™ in which critics like Alan Sokal (he of Sokal affair fame) and Paul Gross accused postmodern theorists of undermining rationality.

Defenders of the situated view shot back that recognising perspective is not the same as nihilistic relativism. Haraway distinguishes her view from ‘anything goes’ relativism by pointing out that claims still have to be accountable to evidence — but one must still recognise that all observers are somewhere.

A related debate turns on perspectivism. Kuhn documented how scientific facts are constructed in specific settings, but scientists often respond that nature ultimately constrains our perspectives. Wishing that the Earth revolves around the Sun does not make it so.

The idea is that there’s tension between realism and constructivism. Contemporary philosophers often occupy a middle ground that acknowledges, while truth isn’t subjective, all access to truth is mediated by perspective.

What about AI?

Large language models draw on billions of tokens of data. Every output is called forth from somewhere, usually conjured up with one eye on a small slice of the internet. Each line has been conditioned by the feedback of human annotators who up-weighted some output and shooed away others.

What we get is a statistical amalgamation of situated viewpoints anchored in very specific conditions. When we request a historical summary of the siege of Khartoum, we’re probably getting the anglophone perspective. In standpoint-theory terms, the model amplifies perspectives that dominate public text and omits those that circulate orally, behind paywalls, or in low-resource languages.

In this sense, the model’s knowledge is deeply situated — though not precisely in the same sense it might be for humans:

• For humans the standpoint is lived. You occupy a body, a culture, a history. Those coordinates shape what you can legitimately claim to know.

• For an LLM the standpoint is inherited. Every sentence reflects the conditions under which it was constructed: data sources, content policies, platform design, technical affordances, and the risk posture of developers.

That makes the model’s outputs situated in origin (they reflect the decisions of its makers) but non-situated in experience (the model itself has no lived point of view). It is anchored in the contingencies of its design rather than in a knower’s embodied life.

Knowledge as power

Taking a leaf out of the pragmatist’s book, one way to make sense of knowledge is to skirt questions about what knowledge is and instead focus on what it does.

This tradition, associated with social theorists like Michel Foucault, describes a great entanglement between what we know and our ability to shape the world around us. It’s a school of thought that turns Bacon’s famous adage, ‘knowledge is power’, upside down.

The move is to take the phrase both literally and critically. Those who define what is known often hold power, and conversely, power structures determine what counts as knowledge.

Foucault famously said that ‘there is no power relation without the correlative constitution of a field of knowledge, nor any knowledge that does not presuppose and constitute at the same time power relations.’

Whenever power is exercised (e.g. the state’s power over citizens, a teacher’s power over students or a doctor’s power over patients), it does so by relying on knowledge (e.g. census data, educational curricula or medical diagnoses). That knowledge in turn reinforces power relations by making them appear natural or necessary.

The problem here is a doozy. If all knowledge is an effect of power, then Foucault’s own analyses are equally compromised. On what grounds, critics ask, can they claim any critical privilege?

Instead, we might say that language contains built-in validity claims that actors must implicitly raise whenever they seek mutual understanding. Power, through say propaganda, can warp those claims — but the possibility of identifying distortion presupposes an ideal of undistorted communication.

It should be clear that we’re not really talking about another type of knowledge here. Knowledge as power isn’t really a fourth species to sit alongside those that we’ve already discussed, but it is useful to think about for getting to grips with how conceptions of knowledge have changed — and how knowledge actually circulates in the world.

What about AI?

I think this type of knowledge, if we can even call it that, is probably the one that gets the most air time in discussions of AI today. At least implicitly, much critical work on AI is essentially Foucauldian. It’s interested in how the AI project uses personal information in service of a developer’s institutional goals.

Of course, this is basically true of any organisation trying to make money — especially those of the information age. More specific criticisms tend to involve studying how AI can actually exert political influence in the real world (though my view is concerns like ‘misinformation’ are overpriced).

This type of analysis also considers the behaviour of the models and how design decisions influence outputs. As in discussions about situated knowledge, content filters, post-training techniques, and other classifiers all encode value judgements about what’s fair game and what’s not.

Reinforcement-learning and policy tuning rely on human raters following a set of guidelines. The result is a model that treats some claims as polite, others as disallowed, and still others as ‘unproven’. All reflect the institutional risk posture of the builder.

That is not to say LLMs are corrupt. Rather, it simply reminds us that the knowledge produced by a model is the outcome of many small (and often invisible) governance choices.

Knowledge as emergence

The final conception of knowledge is the most recent. It holds that knowledge is an emergent phenomenon produced by interactions and networks. This view says that knowledge, whether in science or society, is the product of distributed systems.

A core idea here is actor-network theory (ANT). Developed by Bruno Latour, ANT suggests that all knowledge is collectively produced. In science, for example, knowledge might emerge from the interplay between researchers, instruments, institutions, and even non-human entities like microbes.

An actor-network theorist would say a claim only becomes ‘knowledge’ when a stable network has formed that supports it. You need others to reproduce your experiment. You need journals to publish it. And you need others to cite it. Until that happens, it’s not proper knowledge.

Closely related is the idea of distributed cognition, a concept from Edwin Hutchins’ study of maritime navigation. Hutchins believed that the ‘cognitive unit’ was the whole system of people, charts, instruments, and communication. Only together, he proposed, did they possess the knowledge needed to move the ship from A to B.

We see the idea of networked knowledge in crowdsourcing. On Wikipedia, for example, no single contributor knows everything — but a large network of contributors can collectively produce an encyclopedia.

James Surowiecki puts forward in The Wisdom of Crowds that under the right conditions, aggregating information from many individuals can yield remarkably accurate knowledge. We need look no further for evidence than prediction sites like Manifold, where averaging many educated guesses often beats individual forecasts.

There are many more examples like this, but for this post I’ll leave you with Andy Clark and David Chalmers’ extended mind hypothesis. This theory reckons that cognitive processes (and by extension knowledge) can extend into the environment. They think that a notebook, for example, is effectively an external memory that we can use to remember important information.

Knowledge as emergence is a cool idea, but — like all of those we have discussed — it is not without some problems.

Remember, in the representational view, knowledge is generally thought to correspond with some underlying truth and rationale to back it up. In a distributed system these two come apart. The network may deliver accurate results, but the warrant is…nowhere.

No single node can supply the justificatory story, and the pattern that does support the answer is often opaque. Because justification is diffuse, there is no clear locus for responsibility or error-correction.

What about AI?

Emergent knowledge most neatly corresponds to AI.

As we know, a language model is a dense network that encodes knowledge distilled from massive reservoirs of information. Nothing in the network is built to store explicit facts. Training simply nudges billions of weights so the system gets better at continuing text.

Out of those adjustments, higher-level regularities surface that allow the model to answer questions, draft code, and translate idioms. But no individual parameter ‘contains’ these abilities. If you tweak a few, the behaviour can re-form along alternate paths.

What the model knows arises as a pattern, one that can only exist based on the whole web of connections. Knowledge, in this sense, is an emergent property of the network’s collective dynamics — exactly the phenomenon systems theorists have in mind.

Consider what a model knows about the City of Light. It will tell you that Paris is in France, but there’s no single internal register that stores that fact.

Instead, what we see is that there are circuits responsive to the string ‘Paris’ and others that correspond to a concept of ‘Frenchness’. But that isn’t the same as the symbolic entry Paris → France. The connection between those two circuits is computed from a mesh of activations and embeddings.

More confusing still is that the same fact is encoded in many routes. Ablate one circuit and another pathway compensates. That redundancy is useful for robustness, but it means the fact isn’t localised in a way that carries its own explicit warrant.

Does AI know things?

Knowledge is porridge. It’s warm, thick, nutritious, and bodily. But it’s also unglamorous. It’s hard to separate into its constituent parts and takes the shape of its container.

Thinking about knowledge shows us the limits of language. Much reading later, the only thing I feel truly comfortable saying about the topic is that precise definitions don’t have much currency here.

That all said, the point of this exercise was to answer a question: does AI ‘know’ things? The answer is yes, so long as you squint hard enough and don’t worry too much about fine-print.

So, here are the scores on the doors:

• Representation: AI represents information as knowledge. It mirrors reality well enough to act, yet because those patterns lack a transparent justification, they stop short of the most severe definitions of representational knowledge.

• Practice: Frontier models replay usage patterns rather than consulting explicit rules. Its know-how is powerful but disembodied, missing the sensorimotor feedback that lets human skill refine itself in the world.

• Situation: Its knowledge is deeply situated, though not like ours. Every answer is filtered through the languages, cultures, and editorial choices made by its makers and embedded in the training data.

• Power: Design choices like alignment tuning and corporate policy dictate which claims the model is allowed to produce or suppress. What it knows is bounded by what it is permitted to say.

• Emergence: LLMs are emergence made manifest. Answers arise from the collective dynamics of billions of small units rather than from any single stored fact. The model’s knowledge, like all networked knowledge, is robust but opaque.

So, AI can know if we take ‘knowing’ to mean reliably producing and acting on patterns that line up with the world. But if our definition must include embodied feeling, self-owned explanation, and an unambiguous locus of responsibility, then we have to think again.

Across all five frames the machine scores a partial hit. It mirrors facts, rehearses practical routines, speaks from a data-bound position, carries its builders’ priorities, and generates answers through emergence.

But when all’s said and done, I find it hard to believe that AI doesn’t know anything. It may not know like we do, it may not know everything, but it does know something.

Subscribe now

In 2018, Henry Kissinger wrote about AI in The Atlantic.

He was concerned about our place in a world filled with machines smarter than we are, and wondered whether their coming heralded the end of days for the long age of rationalism.

‘What,’ he asked, ‘will become of human consciousness if its own explanatory power is surpassed by AI, and societies are no longer able to interpret the world they inhabit in terms that are meaningful to them?’

Kissinger is describing a kind of extreme deskilling, one that eventually atrophies the human capacity for reason. In a maximalist version of this scenario, most science is conducted by AI and human researchers become a sort of priestly class charged with interpreting the machine.

The idea is that AI is the final product of the Enlightenment, one that eventually succeeds so thoroughly in harnessing its ideals that it eats the project from the inside out. I’ll write about the ‘new Enlightenment’ another time, but in this post I want to focus on AI (by which I mean frontier LLMs) as the heirs to the age of reason.

When I think about AI, I don’t think about science. I certainly don’t think about rationality. I think about the weirdness of systems that work for reasons we don’t quite understand. I think about odd exchanges with Claude 3.5 Sonnet and the struggle to figure out what is happening inside the models.

I see AI as a clever but erratic organism, one much better at making connections between philosophical ideals or winging film criticism than at assisting in experimental research. You can see this in places where AI is used most: writing, art, coding (that’s creative, isn’t it?) and companionship.

Everyone knows the models are good fun and useful for the right things, but even the most confident user is aware that they aren’t the most rational constructs. They give you different answers every time, are liable to go loco, and occasionally make stuff up.

That doesn’t sound like a successor to the Enlightenment to me. No, AI is the descendant of another great tradition. One that prized intuition over logic, mystery over method, and the sublime over the systematic.

AI is a Romantic project.

Deep yearning

The Romantic movement emerged in Europe in the late 18th century. It began in literature and philosophy before eventually finding a home in music, painting, and politics. A reaction to the hard-headed rationalism of the Enlightenment, at its core was a renewed attention to imagination and individual spirit.

Romantic thinkers saw the world as alive. Nature, with its own moods and meanings, became a source of wisdom. Poets like Wordsworth thought there was an underlying truth in the natural world that rationalism could not account for. Philosophers like Schelling believed the cosmos itself was an expression of living thought.

These images were the product of what the historian Eric Hobsbawm called the dual revolution: the political upheaval of the French Revolution and the economic transformation of the Industrial Revolution. Romanticism emerged in their wake. It looked for coherence in a world that felt unstable amidst the flowering of new forms of cultural, political, and economic life.

The Romantics turned to ruins and folklore, to distant lands and ancient texts. It was a project about longing, about what it feels like to look for something that you can’t quite find. The Germans might call it sehnsucht: a yearning for the indefinite.

In painting, the search looked like stormy landscapes, crumbling abbeys, and wandering figures made small by nature. In music, it took the form of swelling emotion heard in the symphonies of Beethoven or the operas of Wagner.

Romanticism lives in the residue of the past. But where other approaches to cultural remembrance like Classicalism sought to rehabilitate, inherent in Romantic thought is a weary acceptance that revival is impossible.

These ideas are those that animate the AI project. Neural networks are, of course, both very large and very difficult to understand. They are systems whose inner worlds resist comprehension, digital simulacra bludgeoned into submission by a conditioning regime that would make Pavlov blush.

They are the product of planetary-scale logistics and magnets for trillions of dollars in capital. The systems are digital reservoirs filled to the brim by half a century of information deluge.

We’re drawn to AI for what it says about our place in the world: we are small.

This is what is happening when you hear AI people talk about building a ‘digital god’ or terrifying ‘shoggoths’. It’s not about religion or transcendence in any formal sense. But it is a reaction against intelligence as optimisation, one that gives voice to the urge to find meaning in that which we can’t understand. It’s a Romantic impulse in that it insists on the unknowable.

A version of the same instinct sits behind how people respond to models like Claude 3.5 Sonnet. The model was pretty good, sure, but the vibes were just…better than the competition. That’s probably the result of clever post-training, but it’s curious that Sonnet 3.7 doesn’t quite strike the same chord.

Or take GPT-2. When it was released all the way back in 2019, it wasn’t especially coherent. And it certainly wasn’t particularly useful by today’s standards. What it did have was what I can only describe as a kind of lyricism. It didn’t sound like a person, but it didn’t always sound like a machine either.

It was Romantic. I’m not saying that without being a little sentimental, but I do think GPT-2 provoked an emotional resonance that newer models struggle to match despite (or because of?) their polish. That sense of intrigue — not knowing what exactly you like about a half-cooked language model — is the same impulse that made the Romantics tick.

Chasing the sublime

This essay opens with Sadak in Search of the Waters of Oblivion. Painted by John Martin in 1812, the piece shows a man scrambling through a mountainous inferno. He’s alone and desperate, set against a backdrop that reminds us what small things humans are.

It captures what Edmund Burke would call the sublime: something massive, obscure, and overwhelming. A thing that is terrifying and awe-inspiring in equal measure. Size is part of the equation, but the sublime is really about the mind confronting something it can’t fully process.

Towering mountains, violent storms, dark nights were all things that overwhelmed the senses to produce a kind of thrilling terror. Later thinkers like Kant suggested that the sublime came from the mind’s capacity to grasp its own failure to comprehend the infinite.

It’s a useful concept for reckoning with AI at the limit. Trained on more words than any single person could hope to read in a thousand lifetimes, today’s giant neural networks are systems that wrinkle the brain. To create one, you use ultraviolet light to fabricate chips with billions of features. Then you drop thousands of them in a data centre somewhere and ask them to multiply matrices until the sand starts to think.

The networks themselves exist as mathematical constructs on top of the material world. They contain trillions of parameters adjusted via gradients that reflect patterns in high-dimensional space. If you printed them out, they would take you years to read. Researchers generally don’t know what function or piece of information certain parameters correspond to.

Interpretability research tries to close this gap by mapping individual neurons or circuits to linguistic, visual, or conceptual patterns. Some progress has been made, with Anthropic recently showing that certain neurons activate in response to concepts like cities or characters. But this is the exception, not the rule.

These are systems that write poetry, draft emails, and explain quantum mechanics whose internal structure resists human inspection. They reflect us in their language, but their information processing mechanisms are not our own. We know everything about AI, except how it works.

Romanticism was a critique of modernity. As factories rose and cities swelled, its thinkers questioned the idea that technological progress always led to flourishing. Despite the marvels of technology, they saw a world in which each day made it harder to be human.

It’s a concern you are probably familiar with, one that has been reimagined and repackaged by critics of modernity for some time. Robert Putnam’s Bowling Alone famously charted the collapse of community in late 20th-century America, showing how people had grown increasingly isolated from neighbours, churches, unions, and civic groups.

A little later, Daniel Rodgers’ Age of Fracture argued that the collective frameworks that once structured cultural thought (stuff like shared social norms and stable points of reference) were losing out to the irresistible power of the cult of the individual.

Technology didn’t cause these problems, but it did give them form. As AI models stand in for conversation, companionship, and other things we lack, it is hardly surprising that they generate such animosity. They are part of the long story of what it means to be human in a world increasingly built by machines.

This anxiety is most keenly felt in disconnect between what AI is and what it does.

We’re talking about systems trained on massive datasets that are capable of superhuman feats of pattern matching. But anyone who’s spent time in a circular conversation with ChatGPT is unlikely to describe it as rational. Not to mention they can be easily jailbroken, much to the delight of prompt engineers and the chagrin of developers.

Romantic technology

This is why AI is a Romantic technology. Because it is vast and resists understanding. Because it is emotionally and aesthetically resonant. Because it replays our fantasies about creation and asks us to reckon with our place in the world.

These qualities are responsible for the allergic reaction that some people have to AI. When you see someone write off the entire LLM project as ‘bullshit generators’, it’s because they were looking for a product of the age of reason but got one from the age of romance.

To get the most out of AI, we need to manage our expectations. Treat large models as dilettantes rather than librarians. Better to accept that their value lies in surprise and estrangement and pair them with the drier stuff that keeps us honest.

Subscribe now

Subscribe now

Casablanca is one of those films whose reputation precedes it. For anyone unfamiliar, it’s about a guy named Rick who runs a bar in Morocco in the Second World War. Rick drinks whiskey, plays nice with soldiers, and complains about his lost love Ilsa.

She turns up in short order, of course, which gives us the classic line ‘Of all the gin joints in all the towns in all the world, she walks into mine.’ But all is not well. Ilsa is married to someone else.

Her new man is Victor Laszlo, a resistance fighter looking to undermine the Nazi war effort. A lot of moping ensues before Rick—who is now hoping for another shot with Isla—eventually decides to help her flee the country with Laszlo.

Casablanca is a good bit of American propaganda. It came out in 1942 having successfully answered a bunch of questions from US officials like: ‘will this picture help win the war?’ Few will be surprised to hear that the Germans are the bad guys who get their just deserts and the Allies are morally upright fellas who come out on top.

The film is a nice example of what happens when a director is asked to toe the line. Rick and Isla, we are told, are madly in love. But they don’t really act like it. There’s almost no physical affection between them, even in the flashbacks when we’re meant to believe they’re head over heels.

That’s because the makers saw no alternative. Since 1934 the American film industry had operated under the Motion Picture Production Code, a set of self-imposed moral guidelines designed to pre-empt federal censorship. Better known as the Hays Code, the system banned profanity, nudity, and the sympathetic portrayal of crime or adultery. Its function was to present a virtuous version of American life.

When the U.S. entered the Second World War, Washington didn’t need to build a censorship apparatus from scratch. Hollywood had already built one for them. Casablanca was produced within that system, which is why Rick and Ilsa’s affair can be implied but never shown. The Hays Code forbade adultery from being depicted positively, which forces the film to turn romance into ambiguity.

To show an affair would be to endorse it, so the love story becomes elliptical. Rick and Ilsa stare at each other like people remembering dreams. Their lines seem half finished. ‘We’ll always have Paris’ works because the audience is only shown the squeaky clean bits of their time in the city of light.

Rick’s penchant for whining meant that Casablanca never did it for me, but it’s hard to deny that—at least with respect to the love affair—the result is powerful. Passion becomes tension and desire becomes sacrifice. Rick, standing on the airfield, lets Ilsa go because the script won’t let him have her.

The Hays Code was a programme of self-censorship, one that Hollywood imposed on itself because it saw no alternative. By the early 1930s, the film industry was facing a storm of moral outrage from church groups, women's leagues, and state censorship boards for corrupting the public. There was too much sex, too much crime, too many independent women.

Federal regulation began to loom after a string of celebrity scandals agitated lawmakers, much like the ‘video nasties’ craze of the 1980s in the UK that I wrote about a few weeks ago. Fearing the worst, the studios decided to take pre-emptive action to keep the government out and the box office open.

Enter Will Hays, a Presbyterian elder and former U.S. Postmaster General. He was respectable, religious, and well-connected. But perhaps more importantly, he was happy to be the face of Hollywood's clean-up operation.

Under his leadership, the industry adopted rules designed to keep films ‘morally wholesome.’ It banned profanity, nudity, interracial romance, ‘excessive and lustful kissing,’ and any portrayal of crime or adultery that might seem enjoyable. By 1934, the studios agreed to bind themselves to a new enforcement arm, the Production Code Administration, which had the power to deny a film its seal of approval. No seal meant no distribution. The government backed off and religious groups declared victory. The studios kept control as long as they told the right story.

The Hays Code smuggled in a particular view of human nature, one that imagined audiences as easily swayed, morally porous, and incapable of dealing with ambiguity. It saw the movie theatre become a moral classroom where crime was always punished and sex always implied. It should go without saying that the authorities always came out on top.

Civil society was in a contest to decide who gets to define decency and on what grounds. That’s why content moderation is rarely just about content. It’s about the worldview on the other side of the filter and the assumptions about what people can handle, what might corrupt them, what kinds of lives are worth representing. Good intentions or otherwise, behind every censorship regime is a vision of what people are and what they are capable of.

From movie to model

The belief that people are deeply impressionable is the same assumption that animates part of today’s panic over AI. Take misinformation. Powered by generative models that create fake news and recommender systems that serve it en masse, AI is ushering in a new age of epistemic insecurity.

Except it isn’t. Despite the fact that it feels plausible (how can it not when AI certainly has the capability to do these things?), pretty much every credible study out there shows that not to be the case. Next time you hear someone call themselves a ‘misinformation expert’, I encourage you to take a look at their data for yourself.

Despite some recent high-profile moves away from moderation, the last ten years generally followed the path laid out by the Hays Code. Control the flow of information. Shape the message. Protect the public from themselves. Epistemic security for all is certainly a laudable goal, but one that history shows is likely to be tricky to successfully legislate for.

The dynamics work differently with respect to recommender systems (more like engines for distribution) and generative models (more akin to movie-making). The latter, which I’ll focus on the rest of this post, uses a whole bunch of mechanisms to shape outputs. Probably the most visible example are filters that screen responses before they are produced, but the entire development process—from pre-training to post-training modifications to alignment mechanisms—all determine how a model can act.

There are lots of good use-cases for these sorts of tools as they relate to shaping model behaviour. Preventing them from helping people construct biological weapons or conducting sophisticated cyber attacks is straightforwardly good. But aside from uncontroversial red lines, questions about what sort of values an AI model ought to abide by are horribly fraught.

What counts as harm is based on a narrow vision of humanity that assumes we’re all psychologically fragile. Should a model refuse to generate content critical of religion? Should it avoid satire? Should it prioritise safety over openness? The problem with answering these questions is that you inevitably disappoint someone. Even attempts at pluralistic alignment tend to work at the group level rather than the personal.

Whether it’s the Hays Code or content filters, every moderation system encodes assumptions about what knowledge is valuable and who can be trusted with it. AI systems generally prize civility over confrontation, consensus over dissent, and safety over ambiguity. The result is a machine flattens reality into something that is acceptable in the broadest possible terms.

In Discipline and Punish, Foucault argued that power works by shaping what’s normal. By teaching us what questions to ask and how they ought to be answered, moderation contains a certain productive logic. The Hays Code produced a moral universe in which sin could be hinted at but never rewarded, where some desires were real but unspeakable.

Normality manifests in the AI project as models that sound like the HR department. That makes a lot of sense when you remind yourself that the goal of developers is to protect themselves, that are they incentivised to make models that don’t ruffle too many feathers. Many decisions are made by small teams under pressure to minimise risk, drawing on institutional reference points that feel legally non-explosive.

The result is a voice that sounds sterile, but its one that is already starting to change. As people begin to actually use AI, the labs are starting to realise that character counts. That was the bet behind models like Claude Sonnet 3.5 (RIP) and to a lesser extent GPT-4.5, which suggests that nobody wants to pay for a model that sounds like its doing corporate mediation.

As science and technology studies thinkers like to point out, power is about how systems are built. Moderation lives in the pipes. It’s in the training data, the fine-tuning, the user interface, and the invisible refusals. Haraway would call it situated knowledge, a particular view of the world that in this instance we might describe as elite, Western, liberal, and professional.

If knowledge is situated, then so is censorship. The Hays Code reflected the anxieties of a particular slice of American society, usually some combination of white, affluent, pious, and powerful. These groups were asserting a claim about who should shape the moral imagination of the nation, one that was eventually formalised by Hays. What looked like universal decency was in fact a tight set of values elevated through institutional leverage.

Better still, the rules were enforced by the studios themselves. Compliance was built into the production process as scripts were pre-approved, endings rewritten, and scenes cut before they were even filmed. It was a funny kind of censorship machine that could ignore overt repression because the system has internalised constraint.

The boundaries felt natural because the people creating the content were also managing its limits. Far from forcing movie-makers to put out bad films, critics generally agree that the Hays Code ran alongside the Golden Age of Hollywood. Some of my all time favourites like All About Eve, The Philadelphia Story, and Gone With the Wind were all made under the Hays Code.

In part, that’s because writers used the rules as a source of inspiration. A murder couldn’t go unpunished, so noir turned fatalism into an art form. Romance couldn’t be consummated, so filmmakers mastered the language of implication. That doesn’t mean censorship is good, but it does mean that old adage—constraint breeds creativity—has something going for it.

We'll always have Paris

There’s a scene near the end of Casablanca where Ilsa says to Rick ‘But what about us?’ He looks at her and replies ‘We'll always have Paris.’ It’s a line that shouldn’t really work, one that ought to feel at best evasive and at worst hollow.

But it lands because it gestures to something we’re not allowed to see. The affair is gone and the love sublimated into memory. It’s a perfect product of the Code: romantic sacrifice wrapped in moral clarity.

Content moderation has never just been about keeping people safe. It’s about who gets to decide what’s acceptable based on a particular view of what people are like and what they need protecting from. That view shapes what gets censored and what gets created.

Like the film studios of the 1930s, today’s model-makers are engaged in a programme of content moderation or self-censorship (delete as appropriate) to protect the public and to protect themselves. Better to impose rules that shape what models can say than risk government intervention.

The Hays Code shows that self-imposed moral constraints, driven by pressure from the right places, can end up defining an entire cultural era. What began as a defensive gesture became a generative force that reshaped the moral atmosphere of American life.

Something similar is happening with AI, except the stakes are higher. These systems are both interactive and intimate. They’re tutors, confidants, creative partners, and companions. That means that every moderation decision, every filtered answer, and every refusal carries emotional weight.

Culture shapes cultural artefacts, and cultural artefacts shape culture. That is as true for Hollywood as it is AI. Both self-censor to avoid the wrath of the state, both mould tastes, and both set the limits of imagination.

The question is who gets to set AI’s rules. Is it some vague sense of ‘the public’ through focus groups and polling? Is it the trust and safety teams inside the labs? Or in the end will it be government? I see three major ways forward, though there are no doubt others:

• Top-down licensing: Governments eventually mandate a safety threshold. Firms compete on performance inside a fixed compliance box, like age ratings for film.

• User-selectable guardrails: A marketplace of ‘safety profiles’ in which you pick your own filter, sort of like Google’s sliders on its Vertex platform.

• Open-weight maximalism: Open models proliferate, guardrails become optional, and governments tighten application-layer controls.

If only a handful of rule-makers define the guardrails, we inherit their biases at scale. If we embrace openness, we allow more genuinely harmful content to slip through the cracks. Even user-selectable guardrails only deal with your own AI content, rather than content generated by others.

Whatever the case, we’re left with a question with no clear answer: who do you trust to hold the pen?

Subscribe now

I’ll also be sending the second edition of the AI Histories series on Friday. These are short pieces (about 1,000 words) that deal with an important moment in AI history. This week’s will be about the man behind genetic algorithms.

Subscribe now

On the night of January 7, 1610, Galileo Galilei walked onto the balcony and looked down the lens of a telescope. Tilting the device towards the biggest planet in the solar system, he spotted three stars near Jupiter and recorded their positions in a notebook.

He looked again for the same stars six days later, but this time their positions had shifted. For Galileo, there was one likely reason for the change: they weren’t stars at all but moons orbiting Jupiter. The Italian astronomer had long bought Copernicus’ theory that our planet was not the centre of the universe. Now he had proof.

The discovery delivered a blow for the Ptolemaic model of the universe, which held that every planet followed a groove in a transparent, concentric sphere around a stationary Earth. In the geocentric system, any new irregularity—be it retrograde motion or varying brightness—was patched with epicycles, deferents, and equants until the system groaned under its own complexity.

When the little fixes became so numerous, the system eventually became too complex to function properly. The episode is a good reminder that faulty systems don’t break right away, that models of the universe are just that: representations. When a system for representing reality begins to learn more from itself than from the world it was built to describe, that’s when you know its days are numbered.

The Ptolemaic model was created by Claudius Ptolemy, a Greco-Egyptian astronomer who lived in Alexandria in the 2nd century CE. Ptolemy synthesised centuries of Greek astronomy into a grand system described in the Almagest, which became the dominant model of the cosmos in Europe and the Islamic world for over 1,400 years.

The Earth held top billing at the centre of the universe, with the Sun, Moon, planets, and stars orbiting it in perfect circles. To account for the looping paths of planets, it introduced smaller circles, or epicycles, that rode on top of larger ones.

But the model was unstable. Each discrepancy in planetary motion prompted astronomers to add another epicycle or tweak the equant. Ptolemy’s framework was flexible, so much so that it basically overfitted the heavens.

The philosopher Thomas Kuhn, who we briefly discussed in last week’s essay, thought that what appealed to Copernicus was the power to jettison these contrivances from his intellectual framework​. Kuhn famously believed that most science operates within a shared system of sense-making that tells us what questions are worth asking and sets basic guidelines for how to answer them.

But over time, if too many results don’t fit the model, confidence in the paradigm begins to wane. This eventually leads to crisis, which may be resolved by a revolutionary project that brings with it a new way of doing things and new problems to solve.

The Ptolemaic system collapsed when its own patch-work of mends had grown so baroque that it no longer looked credible. Only when those ‘fixes’ piled up did the older heliocentric schemes finally seem the more elegant alternative.

Illusionary collapse

Model collapse is one of those terms that people love to say. It’s often used erroneously, but for now I’ll refer to the definition formulated by Shumailov and colleagues in their 2024 paper:

‘Model collapse is a degenerative process affecting generations of learned generative models, in which the data they generate end up polluting the training set of the next generation. Being trained on polluted data, they then mis-perceive reality.’

The research caused something of a stir last year when it found that as models retrain on their own outputs, they lose contact with the true data distribution. For language models, they found the ‘perplexity score’ of a model (in this case a metric evaluating the likelihood of a sequence of words) worsened significantly when trained on the outputs from a previous version.

According to the group, while the original model fine-tuned with real data achieves a perplexity score of 34, the perplexity score for models trained for five epochs on generated data increases to between 54 and 62. (A higher figure indicates worse performance because the perplexity score essentially measures the inaccuracy of predictions.)

In other words, collapse happens when models forget the variability of reality and converge on a single point​. Nature News called it a ‘cannibalistic’ phenomenon that spelled trouble for large models trained on synthetic data. Models pollute the internet with low quality data, which is fed back into the next generation as part of the training process.

But the devil is in the details.

The researchers trained on 90% synthetic data and 10% original data for each new generation of training. Compare this approach with a paper from Stanford in which synthetic data is added incrementally over time. In the first round the model uses no synthetic data, half of the data is synthetic in the second round, two thirds is synthetic in the third round, and three quarters is synthetic in the fourth round.

Using this method, the Stanford group found minimal evidence of model collapse. As a result, the authors said that the results ‘demonstrate that accumulating successive generations of synthetic data alongside the original real data avoids model collapse.’ Given that real usage is based on accumulation rather than replacement, the upshot is that model collapse probably doesn’t spell doom for AI’s prospects.

There’s a bit in Robert Zemeckis’ Contact where Jodie Foster tries to explain to Matthew McConaughey why his faith in god is misplaced. She evokes the philosophical precept ‘Ockham's Razor’ to which he replies ‘Hockam’s Razor? It sounds like a slasher movie.’

The idea is as well known as it is straightforward: all things being equal, the simplest explanation tends to be the most likely. It takes its name from William of Ockham, a 14th-century Franciscan friar who argued that the Scholastic method had become entangled in abstract distinctions. At the University of Oxford, Ockham found that theological debates often revolved around arguments that seemed more concerned with internal consistency than with observable reality.

In the Scholastic tradition of his time, philosophers engaged in elaborate debates about the nature of universals that he believed led to increasingly complex and self-referential arguments. Ockham didn’t care if ‘redness’ was a quality that existed independent of objects, or even if ‘humanity’ was a real essence shared by all people.

What he cared about was the specific red apple or the individual human being. The rest of it was decadent self-reference.

To convince his contemporaries, the English friar argued that many of the complex distinctions made by his peers were unnecessary. His insistence on paring down explanations to their bones was a response to the recursive character of Scholastic debates, which often built upon layers of prior commentary without re-evaluating foundational assumptions.​

The Scholastic method had become so naval-gazing that it risked losing touch with the realities it aimed to explain. Ockham's push for simplicity was an attempt to recalibrate the intellectual model by grounding it in observable phenomena and logical clarity.​

Ockham's frustration was about systems losing contact with reality. He saw a tradition that had become so fluent in its own language that it forgot what the words were for.

You can see a similar thing happening in art, where critics level that the most dreaded of artistic put-downs: derivative.

Take Parmigianino’s Madonna with the Long Neck, which opens this essay. Completed in 1540, the painting shows us the Virgin Mary with an unnaturally elongated neck, slender fingers, and an elegant posture that defies anatomical correctness.

Parmigianino's work parodies the compositions of his predecessors like Raphael and Leonardo da Vinci. A reaction to the naturalism of the High Renaissance, it distorts its subject to create a sense of elegance that transcends the real world.

Madonna with the Long Neck is an example of Mannerism. Where Renaissance masters like Leonardo, Raphael, and Michelangelo aimed for harmony, proportion, and balance grounded in nature, Mannerist painters embraced distortion, exaggeration, and artificiality.

I think of Mannerism as a kind of recursive practice, an early example of art reflecting on art. Mannerist painters quoted gestures, borrowed poses, and echoed compositions from their predecessors. Often for the better but sometimes for the worse.

For Parmigianino and his contemporaries, the visual grammar of the Renaissance had become so well understood that it was no longer a guide to seeing the world. Instead, it was a set of conventions to be played with, warped, and pushed to the edge of recognisability.

Some critics at the time were unsettled. Giorgio Vasari, a contemporary of Parmigianino’s, admired the Italian’s work. Others found the painting ‘mannered’ in the pejorative sense in that it was stylised for its own sake.

The line that Mannerism walks is the one between homage and parody and refinement and artifice. It’s a reminder that the boundary separating art and slop, original and derivative, is obvious to some but invisible to others.

The feature map and the territory

The French philosopher Jean Baudrillard famously described the modern world as one of simulacra in which representations take on a life of their own. In Baudrillard’s hyperreality, models circulate without stable points of reference and the map precedes the territory.

What he meant was that signs point only to other signs. The meaning of a photo, an advert or a film is borrowed from other media. We consume representations of the world and then representations of those representations. In Baudrillard’s words, the ‘hyperreal’ is when the simulation is more dominant than the real thing.

Like most people familiar with the Frenchman, I find Baudrillard at best head-spinning and at worst frustrating. But his work is useful to understand the relationship between model and meaning, copy and original. It helps us see that, like the Ptolemaic model, imitation looks like stability right before it leads to collapse.

Part of the problem concerns the relationships within a system. Ian Hacking’s ‘looping effect’, for example, shows how feedback moulds relations to allow original category (A) to become perceived category (B) solely through the actions of those who perceive it as (B).

Hacking noted that when people are labelled (say, with respect to a psychiatric condition), they may alter their behaviour in light of that classification. As a result, ‘categorizing people opens up new ways to think of themselves, new ways to behave, and new ways to think about their pasts’.

The very act of modelling changes the modelled. In economics, George Soros has made the same point under the name of reflexivity wherein market participants’ beliefs not only reflect but shape economic fundamentals. The man who broke the Bank of England argues that financial bubbles arise because forecasts influence prices, which alter the reality those forecasts tried to predict.

The point is that knowledge is recursive. A model generates signals and those signals influence modelling. Without a firm anchor to direct experience, the loop begins to tighten. First a little. Then a lot.

There are two ways that people generally describe model collapse in AI. The first, explained above, deals with training-time collapse. It’s about what happens when you repeatedly train a model on its own outputs (or the outputs of other models) over and over again.

The other, which is not actually an instance of model collapse, deals with inference-time feedback. A popular recent example of this idea can be seen in the picture at the beginning of this section. It shows an experiment in which a model is asked to recreate an image exactly as it appears. Seventy-five times later it bears no likeness to the original.

Some commentators reckon inference-time feedback spells doom for the AI project. They think that because these examples show that AI generates sloppy outputs when prompted with originals, the entire internet will be flooded with low quality training material, and subsequently the next generation of models is doomed.

But as we know, if each new model is trained on a combination of human-generated and synthetic data, collapse can be arrested as grounding puts a lid on the feedback loop. If AI-generated data wholly replace human data, the model becomes something like a funhouse hall of mirrors. But if we keep accumulating real text or images, outputs stay solid​.

Both things going on here—the conflation of inference-time feedback and training-time collapse, and the weird ideas about how developers actually make models—point to the idea that talk of collapse is itself self-referential.

The claim becomes a kind of collapse in its own right. It’s recycled, repeated, and stripped of nuance with each iteration as it bubbles up the surface of your LinkedIn feed. People cite the same handful of examples, amplify each other's anxieties, and forget to check whether those fears still correspond to reality.

The Ptolemaic problem

By 1613, Galileo had seen the new world and embraced its implications. Writing with dramatic flair, he scolded his opponents for clinging ‘with such obstinacy in maintaining Peripatetic conclusions that have been found to be manifestly false’.

Galileo’s practical experiments connected Copernicus’s theory to an underlying physical reality. Ptolemy’s cosmos was not a final scientific truth, but it was a necessary mythology to make sense of the cosmos.

This example and others tell us that model collapse is cultural pathology. It emerges when explanatory frameworks (be they cosmological, theological, artistic, or computational) become recursive without recognising their own closure. In each case, the signs are the same:

• In ancient astronomy, the Ptolemaic system became so complex, with epicycles stacked on epicycles, that it eventually collapsed under the weight of its own internal logic.

• In medieval theology, Scholasticism spun increasingly subtle distinctions in Aristotelian metaphysics, which reformers diagnosed as a recursive loop detached from experiential grounding.

• In Renaissance art, Mannerism mimicked and magnified the styles of the High Renaissance. It turned the visual language of naturalism into stylised abstraction that some said was too far from the world.

• In modern computing, model collapse is a fear born from a similar structure. It holds that AI models trained on AI outputs will spiral into self-reference, losing the diversity and grounding once present in human data.

Collapse is about forgetting what the model was originally for. A model breaks when its internal structure becomes so dominant that contradiction, novelty, or reality itself can no longer puncture it.

This is why model collapse is unlikely to pose trouble for the AI project. Developers know what the feedback loop for building resilient systems looks like. They are aware of the the pain points, the need for grounding mechanisms, and the most appropriate way to mix synthetic and real examples to avoid self-reinforcing drift.

Collapse is about a failure to distinguish between imitation and insight, reflection and source, and map and territory. It happens when we forget. When we forget to check our models against the world, when a system forgets its purpose, and when we forget to question our assumptions.

Subscribe now

Emily Dickinson paid attention.

The American poet, who lived most of her life in near-seclusion in New England, wrote nearly 1,800 verses. The collection is about the small stuff. The slant of afternoon light, the tremor of breath, a blade of grass minding its own business.

Dickinson’s work shows us that the world is ripe for remaking. In many of her untitled poems, she rearranges a profound moment around a tiny slice of the ordinary. In one well-known example, Dickinson writes that the last thing one unfortunate subject hears is the buzz of a fly’s wings before their soul leaves their body:

I heard a Fly buzz - when I died -
The Stillness in the Room
Was like the Stillness in the Air -
Between the Heaves of Storm -

The fly reminds us that the mind has a mind of its own, that even the important bits sometimes play second fiddle to the things that shouldn’t matter. That’s the power of attention, the thing that lets us make our own worlds and live inside them.

Attention has been a spotlight, a filter, a resource, and a currency. It’s a shape-shifting concept that connects monks and mindfulness to search engines and self-driving cars, one whose latest afterlife is a mechanism for information processing.

Take an image recognition system like a convolutional neural network (CNN). A CNN works by sliding small filters, which you can think of as screens, across an image. At each step, the network activates certain neurons when a particular feature is detected (like edges or textures) while ignoring others.

Each cluster of neurons responds to only a portion of the whole picture. With enough layers, these local features are combined into more abstract representations: a curve becomes a nose, a circle becomes an eye, and eventually the system recognises a face.

In machine learning as in the brain, understanding is compositional. We can’t attend to everything at once, so we choose to work locally. We obsess over a paragraph, a turn of phrase, or a figure in a painting. The more intense our focus, the more meaningful the pattern.

The literary critic I.A. Richards called this style of analysis 'practical criticism’. Richards gave poems to students without any contextual clues, reporting the results of a 1929 experiment designed to encourage focus on 'the words on the page' rather than preexisting beliefs:

For Richards this form of close analysis of anonymous poems was ultimately intended to have psychological benefits for the students: by responding to all the currents of emotion and meaning in the poems and passages of prose which they read the students were to achieve what Richards called an 'organised response'. This meant that they would clarify the various currents of thought in the poem and achieve a corresponding clarification of their own emotions.

Richards’ approach relates to the idea that knowledge is situated. An important bit of the science and technology studies canon, the point is that one can never see the whole system — only a perspective within it. All vision is partial perspective.

That is true for people, organisations, and systems. It’s also true for scientific discovery, a point made clear in Thomas Kuhn’s famous work on the formation and stabilisation of scientific paradigms. We might say that pre-paradigmatic science is chaotic, but once we get a filter (in this instance, a scientific framework) we start to see things that we couldn’t before.

Paradigms are ways of selectively attending to data. They obscure the full picture so we might see more clearly, all the better to help us sort signal from noise. Like image recognition systems or Dickinson’s poetry, paradigms work because they pay attention to the right things.

Divining attention

Attentive awareness goes back some. One could credibly make the case that it allowed early humans to make meaning and abstract the world around them in useful ways. After all, the hunter-gatherer’s life depended on noticing the flicker of prey in the undergrowth, subtle shifts in the wind, and tracks barely pressed into mud.

In the long run, the ability to focus supported the emergence of complex rituals, artistic practice, and forms of mysticism. The French anthropologist Marcel Mauss said that ‘underlying all our mystic states are corporeal techniques, biological methods of entering into communication with God’.

William Golding’s novel The Inheritors deals with the collision between homo sapiens capable of selective attention and Neanderthals who live moment to moment. While the book anachronistically gives homo neanderthalensis religion, it does a wonderful job at describing how beings with limited attentive capacity might have made sense of the world.

Attentive practice and mysticism became more important as humans traded hunting for agriculture. In ancient Egypt, priests observed the movements of stars and sacred rites with exacting focus. They believed that attention maintained the balance between order and chaos.

In the Vedic traditions of early India, attentiveness underpinned the meditative practices aimed at perceiving the hidden unity of Brahman, the ultimate reality. Across the Mediterranean, the mystery cults of Orphism taught that salvation depended on vigilant self-awareness during life and death.

Ancient Greek philosophy eventually developed its own form of contemplative vigilance. The Stoics called it prosoche, a steady stream of active attention for living in accordance with the rational logos that pervades the cosmos. To attend was to consciously participate in the divine structure of reality.

Early Christian monastics saw attentio (or nepsis, meaning ‘watchfulness’) become a virtue. The first generations of Desert Fathers, a group of ascetics who lived in Roman Egypt, singled out attentiveness as a fundamental Christian moral quality. For the Desert Fathers, prayer itself was a pure form of attention to the divine.

On the other side of the world, Buddhist contemplative traditions fashioned the training of attention into a sophisticated discipline. The Buddha’s teachings about mindfulness (sati) and concentration (samādhi) can be read as prescriptions for holding one’s attention in the present.

Like their Buddhist counterparts, the desert ascetics emphasised lived practice over theoretical knowledge. One trained attention through recitation of the psalms; the other used the sutras. Both traditions recognised that verbal practice served as scaffolding for attentional states.

These traditions understood that the untrained mind wanders endlessly, and prescribed attentional discipline as the antidote. Desert Father Abba Moses’ advice to ‘sit in thy cell and thy cell will teach thee all’ sounds eerily similar to the Buddha's emphasis on solitary meditation as a vehicle for insight.

By the early Middle Ages, attention began to look something like a formalised discipline. What had once been the preserve of solitary ascetics or wandering monks was pulled into the orbit of the institutions that governed religious and intellectual life.

In the monasteries of medieval Christendom, attentiveness became a rule-bound discipline, essential for the copying of sacred texts. In the Islamic world, scholars preserved and systematised ancient knowledge through attentive reading and commentary. Across Europe, the rise of Scholasticism gave new life to on ordered training of the mind and the belief that attentiveness lit the path towards divine truth.

The Italian priest Thomas Aquinas treated attentio as a crucial operation of the soul, the means by which intellect and will could be properly directed toward God, truth, or moral goods. Paying attention was to marshal scattered faculties into a disciplined focus. It was in some ways a rational act, one that represented victory over the unruly appetites that tempted the soul.

This rational-ish style of attention gradually evolved into an intellectual virtue at the heart of Renaissance humanism. Scholars like Erasmus urged readers to cultivate active attentiveness in an age when the printing press flooded Europe with classical and scriptural texts. To read well was now to sift wisdom from error, to attend to meaning rather than memory.

Erasmus urged readers to dwell, discriminate, and refine the mind through the careful discipline of reading and thinking. A century later, Descartes pushed the logic of attentiveness further by arguing that attentive scrutiny was the foundation of certainty.

His method of doubt, famously outlined in Meditations, demanded a focusing of the mind’s gaze away from the noisy flux of the senses and towards only those ideas that could be grasped with absolute clarity. Once spiritual and later humanistic, attention was becoming the instrument by which knowledge could be assembled from the ground up.

In the eighteenth century, Immanuel Kant built on Gottfried Leibniz’s concept of apperception, which deals with how the mind becomes aware of its own perceptions, to argue that coherent experience depends on the active work of the mind. For Kant, the mind bound perceptions together into a unified world through an original act of synthesis.

Kant is an important figure in our narrative because he shifted the philosophical terrain from the contents of thought to the structures that made thought possible. It might sound a bit niche, but this distinction encouraged others to view the mind as an active organiser of experience rather than a passive receiver of impressions.

Rethinking attention

Wilhelm Wundt founded the first psychology lab in 1879. As a young man, the German struggled with nervousness and daydreaming. He was reportedly prone to losing focus and drifting into reverie.

Wondering why he struggled to stay on track, Wundt set about to study the psychological dimensions of attention. He began with Leibniz’s apperception, which he treated as the voluntary direction of attention by which sensations were selected, organised, and made conscious.

Wundt was a Kantian in that he believed that experience depends on the active organisation of perception. But where Kant had treated this as a necessary and universal structure of mind, Wundt focused on the elective aspects of attention. He wanted to know how we selectively direct focus, how we amplify or suppress sensations, and how these choices could be measured experimentally.

Around the same time, physicist-turned-physiologist Hermann von Helmholtz performed a simple but important experiment. Fixating his gaze on the centre of a briefly illuminated array of letters, he found that he could covertly shift his attention to a different part of the array. Helmholtz could make out letters in this region while the rest remained a blur.

The central insight was that attention could act independently of eye movement. Attention, as it turned out, was a distinct cognitive faculty, one capable of selectively enhancing experience from within.

Around a decade later, William James confirmed attention’s role in the new psychology. In The Principles of Psychology (1890), James described attention as the mind’s act of taking ‘possession of one out of what seem several simultaneous objects or trains of thought’ emphasising that ‘focalisation, concentration, of consciousness are of its essence.’

He argued that focus is a process of active selection, one that ‘implies withdrawal from some things in order to deal effectively with others.’ For James, attention was the essential mental act. It made order from chaos, clarity from confusion, and identity from experience.

James had placed attention at the heart of mental life, but within a generation, psychology’s centre of gravity had shifted. In the early twentieth century, the rise of behaviourism relegated attention to the margins of scientific respectability. Figures like John B. Watson insisted that psychology must concern itself only with what could be directly observed, namely external stimuli and behavioural responses.

In line with the new paradigm, attention was redefined as the observable orientation toward certain stimuli rather than others. The idea of attention as an active, inner force shaping consciousness—so central to Wundt and James—all but disappeared for several decades.

But it wasn’t to last. A new generation of researchers argued that psychology could not make sense of complex phenomena like language, reasoning, or memory without grappling with internal mental processes.

Known as the cognitive revolution, this movement reimagined the mind as an active processor of information capable of selecting, storing, retrieving, and manipulating inputs much like a computer. In this new framework, attention returns as a central object of study. No longer dismissed as unmeasurable, it was now seen as a mechanism for managing the mind’s limited processing capacity.

Psychologist Donald Broadbent gave the new cognitive psychology one of its first working models of attention. In 1958, drawing on wartime research with pilots and radar operators, he proposed that the mind uses an early-stage filter to manage the flood of incoming information. Like a gatekeeper at a switchboard, Broadbent hypothesised that an attentional filter allows one stream of sensory input through for conscious processing while blocking the rest.

This neatly explains why, when listening to two people speak at once, we can follow one conversation and tune the other out. Broadbent's model was crisp and mechanical, treating attention as a bottleneck tuned to a selected channel. It was elegant, and for a time, dominant — but brittle.

In certain listening studies, participants could sometimes pick up personally relevant information like their own name in an unattended conversation. This suggested that unattended material was not completely blocked. Another experiment went further, demonstrating that people could unconsciously piece together meaningful phrases from words split across different audio channels.

Building on these tests, English psychologist Anne Treisman proposed that attention was not a screen so much as a volume control where unattended inputs were weakened but not completely eliminated. Meaningful information could still break through if it crossed a certain threshold of relevance.

Treisman later turned her hand to visual perception, developing what became known as Feature Integration Theory (FIT). She showed that basic features like colour, shape, and orientation are first registered automatically (and separately) by the visual system.

The theory held that without attention the brain can conjoin features incorrectly— seeing, for instance, the colour of one object attached to the shape of another—in a phenomenon known as illusory conjunctions. According to FIT, attention was a necessary process for binding separate features into coherent perceptual objects. Without it, the visual world would fragment into disjointed colours, shapes, and patterns.

During these decades, psychologists began to develop new metaphors to capture how attention behaves. Michael Posner famously likened attention to a movable ‘spotlight’ that selectively enhances processing wherever it points. Others proposed a ‘zoom lens’ model suggesting that we can widen or narrow the scope of our focus by trading breadth for resolution.

The metaphors masked a deeper shift. Attention was less static filter than dynamic resource that the mind could steer, widen, or narrow. In the cognitive view, attention was an active force that regulated the flow of information moment by moment rather than holding focus behind a single gate as in Broadbent’s model.

Daniel Kahneman carried forward the interpretation of attention as an active process, but reframed it in terms of limited mental resources. In his 1973 book, the great psychologist likened attention to a finite pool of energy that could be allocated flexibly across tasks. Performing two tasks at once was difficult if they drew heavily on the same attentional reserves, but easier if one was automatic or if they tapped different cognitive systems.

By the late twentieth century, attention could be seen at work. New techniques like the electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI) allowed researchers to connect attention to brain activity. Neuroimaging studies identified regions like the intraparietal sulcus and the prefrontal cortex that acted as ‘searchlight operators’ that directed focus and regulated the flow of information.

Neural data suggested that attention was not a single mechanism but a collection of related processes like orienting to stimuli, filtering out distractions, sustaining focus over time, and switching flexibly between tasks. Attention had become a material process, one that would influence technology and our relationship with it.

Mechanising attention

Herbert A. Simon, a giant in the history of AI, said that ‘In an information-rich world, the wealth of information means a dearth of something else…what information consumes is the attention of its recipients. Hence a wealth of information creates a poverty of attention.’

For most of human history, information had been scarce and attention plentiful. Now, information was everywhere, and attention had become the limiting factor. In their 2001 book, The Attention Economy, Thomas Davenport and John Beck argued that this shift was creating a new kind of economy in which value was measured by the ability to capture and hold human attention.

The attention economy concept, though related to the psychology of attention, is not identical to it. In cognitive science, attention describes the selective processing of information under conditions of limitation. In the marketplace, attention is measured externally through clicks, views, and time spent.

The brain’s limits turn attention into a zero-sum resource. Over time, psychological findings about attention—its spans, its bottlenecks, its failures—have been absorbed into everyday language. What was once a description of mental effort has become a common framework for parsing a mode of living under the weight of information.

There are of course two parts to Simon’s famous quote. We all remember the bit about attention, but what may prove more significant is the sheer volume of information being produced. He didn’t know it then, but information would become the raw material for a new generation of machine learning systems inspired by attention itself.

In the mid-2010s, machine learning researchers began using the term ‘attention mechanism’ to describe a method for selectively processing information. Rather than treating all inputs equally, these systems could focus on the most relevant parts of the data. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio introduced the first influential version of this idea in neural machine translation, allowing models to dynamically align with specific words in a source sentence as they generated translations.

Instead of compressing an entire sentence into a fixed representation, Bahdanau’s model saw the system search through the source sentence during translation. The model could attend to different parts of the input at each step, much as a bilingual speaker might glance back at the original text to refine a translation.

But the major moment for attention, as all AI watchers know, came in 2017 when Google described the transformer architecture built around the famous self-attention mechanism. Where Bahdanau’s model used attention to improve a sequential system, transformers opted to discard recurrence altogether. Instead of processing information step by step, the architecture used multiple attention mechanisms in parallel to compare every part of the input to every other part at once.

In a transformer, every word in a sequence can attend to every other word through learned weightings. For each word being processed, the model calculates attention scores between that word and all words in the input. It basically asks ‘How much should I pay attention to word X when processing word Y?’

This approach yields an attention map that tells us which parts of the input are most relevant to each word being processed. By using many heads that each focus on different aspects of the input, the model can pick up multiple kinds of relationships at once such as syntax or semantic context.

The result is a rich representation of language (and as it turns out, probably a lot more than that). To put it simply, language models amplify important signals and suppress less important ones. Their use of attention has even prompted loose comparisons to the way neural circuits in the brain manage focus and selection.

That isn’t to say that language models are brains, but rather that the convergence between the two reflects shared constraints if not a shared nature. Faced with too much information and finite processing power, both minds and machines must solve the same underlying problem: how to select what matters.

Attention is a product of limitation. Whether in the prayers of ascetics, the experiments of early psychologists, or the designs of machine learning engineers, the problem is the same. It’s the one about how to find coherence in a world made up of more than can be grasped.

If the story of attention tells us anything, it’s that all seeing is partial. The mind knows by focusing. Science progresses by filtering. Machines learn by selecting. No perspective, human or otherwise, can capture the whole. To attend is to miss almost everything, but that’s the whole point.

Emily Dickinson once wrote that ‘a letter always feels to me like immortality because it is the mind alone without corporeal friend.’ What she meant was that letters offered a way for the mind’s expression to endure on its own, that they exist as fragments of the self that outlast the living.

In her poems, attention works in much the same way. A single moment, chosen and held, becomes a whole world. The mind’s spotlight falls where it falls, whether on the wings of a fly, a band of light, or a blade of grass.

Attention is not what limits our experience of the world. It’s what makes experience possible at all. Without the narrowing there would be no coherence, no meaning, and no world to perceive.