Contra Chiang on machine consciousness

If you think you understand AI consciousness, then you don't understand AI consciousness

See also Bentham’s Bulldog, “Ted Chiang Is Wrong About AI Consciousness”, and Rob Wiblin, “Ted Chiang is Wildly Overconfident About AI Consciousness.”

Ernst Haeckel, "Hexacoralla," Kunstformen der Natur (1904)

Ted Chiang is one of my favourite authors. I was a philosophy PhD student when I first came across his novella The Lifecycle of Software Objects back in 2011, and it completely blew me away in its nuanced treatment of the identity, moral status, and autonomy of advanced (and possibly sentient) machines.

By contrast, I’ve found his non-fiction writing on contemporary AI systems quite frustrating. Shortly after the release of ChatGPT in early 2023, for example, in a comment piece in The New Yorker he characterised the model as a “blurry JPEG of the web,” and suggested that LLMs could at best provide paraphrases of information that was already out there, with limited utility for true creative work. While the piece correctly anticipated some things, including the torrent of slop that LLMs have enabled, other predictions haven’t held up.

Most notably, Chiang predicted that labs would carefully exclude LLM-generated text when training future models, on the grounds that training on model outputs would be like photocopying photocopies. As it happened, within a couple of years LLM-generated outputs were being routinely used for both pre- and post-training of frontier models, with painstaking benchmarking showing that, used appropriately, they can improve performance.

More broadly, the blurry JPEG analogy was an unhelpful one, and anyone who anchored on it too strongly would have been poorly placed to anticipate what happened next, from reasoning models and IMO gold medals to novel mathematical proofs. A JPEG is a static and compressed artefact; LLMs, by contrast, are deliberative and generative systems capable of delivering novel insights and analysis and producing the very data that improves their successors.

Chiang’s AI skepticism has recently spilled over into the domain of consciousness, very much my home turf. In a new signed essay in The Atlantic, he confidently declares that contemporary AI systems are not conscious, and insists that suggesting otherwise is an error of “titanic magnitude.” These systems are just “cleverly disguised examples of sentence continuation,” and that to even be open to the possibility of consciousness in LLMs is “the same as being open to the possibility that Microsoft Word is conscious.”

These are extremely strong claims to make about a phenomenon as confusing, complex, and contested as machine consciousness. While I think it’s unlikely that current models are conscious (at least in any form meaningfully similar to humans or animals), I certainly don’t think it’s a basic error or confusion to suggest otherwise, and I’m in good company among cognitive scientists and philosophers of mind.

So does Chiang bring the receipts to back up his bold claims? Unfortunately, I think the answer is a pretty clear “no”, and he instead serves us up a buffet of rather stale and in some cases outright confused arguments.

Chiang’s essay doesn’t follow a traditional philosophical argument, but I’ve grouped his claims about machine consciousness into four main clusters, none of which come close to supporting his bold headline, as we’ll see. I have a bit more sympathy for some of the ethical and political musings he closes with, and I’ll return to these at the end of this piece, but even there I think he gets way out over his skis and makes some overly strong (not to mention uncharitable) assertions.

(1) Most Things Are Lots Of Things

At the core of Chiang’s critique is the idea that we fundamentally misunderstand what LLMs are: he sees users as tricked by their linguistic capacities into thinking that more is going on than meets the eye. Rather than being minds (or even proto-minds), “LLM conversations are cleverly disguised examples of sentence continuation”, a “predictive text game” from a machine that “generates only one word at a time.”

I should note that in one respect I agree with Chiang: there are real dangers of anthropomorphising LLMs and treating them as more human than they really are. Contemporary AI systems are anthropomimetic, not truly humanlike, and differ dramatically from us in mechanism, architecture, and embodiment.

But this isn’t a royal road to dismissal of AI consciousness. Almost anything can be described at multiple levels. We might variously describe a human baby as a mass of fermions and bosons, a collection of proteins, a conglomeration of cells, a biological organism, the child of Mr and Mrs Jones, or a young Englishman. None of these descriptions are in competition with each other, but operate at different levels of analysis.

The same is true of informational systems, and this point was famously made by British cognitive scientist David Marr in his landmark work Vision (1982), published posthumously following his untimely death. When we look at something like human vision, we can describe it in terms of its implementation (neurons, synapses, and so on), its algorithmic features (how it computes properties like light, shade, and contour), and its underlying computational function. Crucially, these levels aren’t in competition: just because a system is a bunch of neurons doesn’t mean that it’s not also doing nearest neighbour interpolation or working out how fast a ball is moving.

This is why characterising LLMs as functionally equivalent to predictive text systems or statistical inference machines and inferring on that basis that that’s all they are is a mistake. It’s perfectly compatible with them being predictive engines that they’re also cognitive agents with beliefs, desires, attitudes, and even - potentially - consciousness.

Scott Aaronson has a useful name for the trick that Chiang is pulling here, namely “Justaism”: LLMs are just next-token predictors, just function approximators, just gigantic autocompletes, just stochastic parrots. But almost nothing is ‘just’ anything: what Marr teaches us is that most things are lots of different things. Just because we’re made out of meat doesn’t mean we’re not also res cogitantes.

There’s another lurking worry here for Chiang’s specific version of Justaism, namely that the predictive capacities he sees as profoundly unhumanlike may not be so removed from algorithmic-level features of our own cognition. Indeed, predictive processing approaches to biological cognition are an extremely popular paradigm in contemporary cognitive science and computational neuroscience, popularised by philosophers like Andy Clark and neuroscientists like Karl Friston. According to these frameworks, our brains are constantly running internal generative models to minimize prediction errors about sensory inputs.If prediction-based generative modelling is central to human cognition, then the mere fact that LLMs are predictive systems cannot by itself count against them as consciousness candidates.

(2) Cognition Lives In Processes, Not Outputs

A second conflation of levels going on in Chiang’s essay comes from his tendency to confuse the outputs of LLMs with the processes that give rise to those outputs. Chiang confidently declares that we can “definitively say that certain things are not conscious, and conversational transcripts fall in that category.” He goes on to assert that being open to AI consciousness is “the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript.”

Of course, this is a strawman of machine consciousness: nobody thinks the transcript itself is conscious, any more than we think human consciousness lives in the sound waves we emit when we talk to each other or the footprints we leave on the beach. Transcripts are public records of what the LLM is doing, and if consciousness lives anywhere, it would be in the dynamic computational mechanisms producing those outputs.

This brings us to Chiang’s insistence that an LLM is a primitive device because it “generates only one word at a time.” While it’s true that language models deliver their outputs one token at a time, it’s not immediately clear on what this has to do with consciousness. Could we not imagine a simple conscious mind that operated one computation at a time?

I assume that Chiang is presumably leaning on this as evidence of a disanalogy between humans and LLMs. But of course, humans emit language serially too; our underlying thoughts may be non-linear and deeply layered, but even we produce out verbal outputs one morpheme at a time (in spoken language, at least). Consequently, the fact that a system’s outputs are serial tells us very little about the depth or complexity of the computational engine running underneath.

If Chiang’s claim is intended to go beyond outputs to the stronger position that LLMs only compute one token at a time, he’s on shaky ground empirically. Recent work in mechanistic interpretability (most notably by Lindsey et al. 2025; see also Maar et al. 2026) has demonstrated that frontier LLMs engage in more holistic problem solving, or “latent planning”. When a model is tasked with writing rhyming poetry, for example, it does not in fact compose the poem one word at a time. Instead, researchers found causal evidence of internal planning circuits that preselect a target rhyming word multiple steps in advance, and then perform “backward planning” to structure the preceding sentence to fit with the rhyming. The outputs might be delivered one token at a time, but they’re composed in a more holistic fashion (my poetic composition tops out at bad limericks, but when I’m coming up with those, I do something rather similar). Of course, this doesn’t show that LLMs are conscious, or even that they plan like humans, but it should shatter the idea that just because they’re outputting a single token at a time that they’re simply thinking one token at a time.

The boldest rhetorical flourish of the piece is Chiang’s asseveration that openness to LLM consciousness entails openness to consciousness in Microsoft Word. Again, it’s not clear what the argument here is meant to be, but I take it that for such a declaration to make sense, it would have to be the case that LLMs have no special features that could potentially instantiate consciousness that aren’t also shared by Microsoft Word. But this is obviously false: Microsoft Word is a passive text repository, while a frontier neural network is a dynamic, multi-layered statistical engine transforming high-dimensional vectors to perform semantic inference. The fact that they’re both systems that produce text is one thing they have in common, but it’s less interesting than their myriad differences. To say that openness to consciousness in one commits us to openness to consciousness in the other is like saying that openness to consciousness in chimpanzees means we must be open to consciousness in slime moulds, on the basis that both are made of biological cells, while ignoring the differences in complexity and behaviour. Similarly, having a non-zero credence in some form of machine consciousness does not conceptually trap you into panpsychism for machines, or into thinking that office software might be conscious.

(Well, with the possible exception of Microsoft Excel, which is admittedly Turing-Complete.)

(3) Vibes Make For Bad Arguments

Chiang’s “Justaist” claims may be flawed in their repeated conflation across levels of explanation, but at least they’re attempts at arguments. By contrast, many of his other positions rely essentially on vibes. When Chiang states that it’s “not plausible” to him that an LLM getting progressively better at roleplay could ever be conscious, this is a simple appeal to personal incredulity. We all engage in a bit of intuition-mongering from time to time, but for an essay that carries the absolute headline “No, Artificial Intelligence Is Not Conscious,” we need more than that.

It’s worth taking a step back here to note that Chiang is not alone among machine consciousness sceptics in relying on gut-checks as arguments. There is a long history of arguments from incredulity in the field, from Searle’s Chinese Room to Block’s Chinese Nation.

I’m an outlier in saying this, but I think this reliance on intuition is a pretty awkward state of affairs for a discipline that takes itself to be engaged in a fundamentally scientific project. In defense of Searle and Block, they were writing at a time while consciousness remained a largely philosophical matter and conceptual analysis was one of the only tools we had at our disposal. However, most contemporary work in the science of consciousness explicitly or implicitly treats consciousness as a candidate natural kind; that is, a deep biological or computational property whose underlying nature we have yet to discover.

In this respect, most researchers hope to figure out consciousness in something like the way we arrived at DNA as the mechanism of heredity: we could see that offspring resembled parents and had good reason to suspect there was some deep mechanism there, but it took almost a century after Darwin’s publication of The Origin of Species to pin down the actual biochemical process. Similarly, we have surface level observations about consciousness (primarily our own experience and its associated neural activity), but we haven’t cracked the underlying mechanism.

If we’re just doing conceptual analysis, we can demarcate the boundaries of our inquiry in advance on the basis of semantic intuition, but if we’re treating consciousness as a natural kind, we have to be open to being surprised. Science is full of cases where reality turned out to be much weirder than we expected, from quantum mechanics to black holes and mitochondrial DNA. Our intuitions give us an initial anchor and explanatory starting point, but they’re a ladder we need to be willing to leave behind.

This is particularly true for exotic forms of consciousness like those which might one day emerge in machines. Why should our visceral intuitions about a completely novel, non-biological substrate carry any ontological weight? Human intuitions about other minds evolved to navigate a hierarchical primate society, highly specialized for reading facial expressions, biological motion, and social cues. When confronted with an architecture that is disembodied, digital, and alien, our ancestral gut is out of its epistemic comfort zone. Trusting our immediate intuitions to deliver a verdict on machine consciousness is about as epistemically sound as trusting an 18th-century farmer’s intuitions to deliver a verdict on spacetime curvature or quantum entanglement.

The alternative of course is to rely on scientific evidence and rigorous scientific frameworks like Global Neuronal Workspace Theory, Integrated Information Theory, Attention Schema Theory, and more. These have big problems, to be sure, but at least they’re better than vibes. And yet Chiang’s essay contains a striking paucity of any consciousness science. Perhaps there’s good reason for this; as Bentham’s Bulldog notes, most leading theories are in fact fairly amenable to the possibility of machine consciousness. Butlin et al. are quite clear in their magisterial report on machine consciousness that “there is a strong case that most or all of the conditions for consciousness suggested by current computational theories can be met using existing techniques in AI.”

In defence of Chiang, he does invoke one leading machine consciousness expert in the form of Murray Shanahan, citing his descriptions of LLM interactions as an engrossing form of “role-play” or collaborative authoring. However, speaking as someone very familiar with Murray’s work, I couldn’t help but raise an eyebrow at Chiang’s intellectual cherry-picking. Shanahan’s broader view on AI consciousness (articulated in works like “Simulacra as Conscious Exotica”) is skeptical of the very framing of the debate that Chiang relies on, namely that questions of machine consciousness involve a mysterious inner light distinct from behaviour and patterns of human-AI relation. As Murray puts it, “We must resist the temptation to ask whether [an AI] is conscious as if consciousness were something whose essence is out there to be uncovered by philosophy (or neuroscience)... [i]nstead, we can ask whether it would be possible to engineer an encounter with the entity, and how our consciousness language would adapt to the arrival of such an entity within our shared world”. This deflationary insight cuts symmetrically for AI enthusiasts and skeptics alike, but Chiang only keeps the side that wounds the former. I suspect Chiang would find that he and Shanahan agree about rather less than he seems to think.

(4) More than is dreamt of in your philosophy

Throughout the essay, Chiang frames himself as the voice of reasonable common sense, dispelling the phantoms of those hapless AI consciousness boosters caught in the throes of anthropomorphism and theoretical dogma. But a closer read makes clear that he’s placing a lot of strong theoretical bets of his own. The first and most glaring of these is his absolute commitment to strict biochemical chauvinism. Chiang writes:

“an emotion such as desperation is inseparable from having stress hormones such as cortisol and epinephrine… having a conscience means feeling sadness or moral repulsion at the idea of taking a certain action, and those emotions entail a physiological response...”

I won’t outright say that Chiang is wrong here - the neuroscience of emotion is almost as contested a field as consciousness - but I’d flag some fancy prestidigitation in how he frames these claims. It is true that our own emotional responses are tightly coupled with the specifics of our physiology: beta-blockers work for panic attacks because they blunt adrenal response, and it’s thanks to cortisol we wake up with a racing heart and a sense of dread (or is that just me?).

However, we shouldn’t confuse the causal role of hormones and neurotransmitters with the psychological events they precipitate: on most views of the role of hormones in emotion, it’s more accurate to say that cortisol triggers and modulates our affective states rather than strictly constituting them.

To illustrate, imagine that in the far future we meet a race of intelligent aliens living in the clouds of Venus. They have a sophisticated society, complex interpersonal relationships, and a rich artistic tradition. However, they’re also profoundly physiologically different from us, with no clear analogues for our endocrine and nervous systems. Nonetheless, our xenoanthropologists go out and live among them and successfully learn their languages (perhaps they even communicate through multidimensional ideograms that we find hard to interpret).

One day, a xenoanthropologist is chatting with one of the Venusians and is startled to hear the alien declare that they are feeling desperation as a result of rejection by a romantic partner. Should the anthropologist dismiss this, on the basis that the Venusian doesn’t have cortisol or epinephrine flowing through their tentacles? Should we say that it’s only real desperation if it comes from the Terran region of the solar system, and otherwise it’s merely sparkling hopelessness?

The point here is not that Venusian desperation is likely to feel identical in every way to human desperation - that would be an improbable functional convergence. But I do take the example to illustrate that our emotion-concepts are not rigidly bound to specific hormones and neurotransmitters, and instead are ultimately grounded in the cognitive and functional role they play. What individuates despair is not cortisol or epinephrine, but shattered hopes.

While I disagree with Chiang that emotions require hormones, I can meet him halfway in the idea that our physiology is in practice at least causally implicated in the precise phenomenological contours of our emotions.

What I find really baffling is his suggestion that this extends to ethical reasoning. He confidently asserts that moral understanding is “necessarily subjective because it relies not just on an individual’s intellectual response to a problem but also on their emotional one.”

This seems to me a much more extravagant claim. While emotions can certainly influence our ethical stances, the idea that all morality relies on hormones is an incredibly radical view. With a single wave of his hand, Chiang takes a stand on one of the oldest debates in philosophy, entirely endorsing a sentimentalist, Humean view of morality. This is a live debate in meta-ethics, but the point is that Chiang presents one highly contested position as settled fact. He writes off the entire Rationalist and Cognitivist tradition - running from Plato right through to Kant - which argues that moral reasoning is fundamentally an exercise in objective, rational justification. If Kant was right that morality is a matter of pure reason, then a sufficiently advanced algorithmic system is precisely the sort of thing that could engage in bona fide moral reasoning. And one doesn’t need to be a Kantian to endorse something like this; many contemporary Consequentialists who feel that morality consists in maximising preference satisfaction or minimising suffering should also be open to the idea that these goals could be independently arrived at by a sufficiently smart machine, emotions be damned.

Meta-ethics is of course a famously baroque and challenging discipline (my Cambridge grad students frequently get headaches when meta-ethics enters the room), and I wouldn’t blame Chiang for having inchoate views on the topic. But he also makes dubious and to my mind straightforwardly false claims about the implications of his position for moral reasoning in LLMs, for example confidently declaring that without emotions, “an LLM can only rephrase expressions of moral reasoning found in its training data.”

Speaking as someone who talks extensively about ethics with LLMs in both a personal and professional capacity, I can confidently say that modern frontier models do not merely copy and paste ethical maxims, but possess the capacity to synthesize, extrapolate, and apply abstract moral frameworks to radically novel out-of-distribution scenarios. If you present a frontier model with a highly bizarre, complex ethical dilemma that exists nowhere in its training corpus (say, an intricate trolley problem involving alien biologies, collective forms of consciousness, and exotic technologies) it will do its best to come up with reasonable answers grounded in normative ethical theories, moral common sense, and arguments from analogy.

Perhaps Chiang would say that this is still generalising from information in its training data and reinforcement learning curriculum (in the case of Claude, determined in part by its constitution). But if we define real moral reasoning so stringently that applying generalized frameworks to new cases counts as merely extrapolation from training data, then I’d also have to flunk the majority of my undergraduate students. We all get our ethical foundations from somewhere, after all, and there’s a reason that Classical philosophers and modern psychologists alike recognise the pivotal role of childhood upbringing in formation of moral character.

I don’t pretend to suggest that debates about substrate independence or meta-ethics or moral psychology are settled; these are live and complicated questions. But Chiang’s entire essay frames AI researchers and open-minded philosophers as naive hypesters falling under the magical spell of an anthropomimetic machine. And yet the reality is that Chiang is the one making repeated enormous leaps of faith, reliant on a highly partisan cocktail of strict biochemical reductionism and sentimentalist metaethics. Bold, contested philosophical positions are perfectly fine to hold, but they ought to embarrass you into at least a little humility. They certainly don’t earn you the right to declare your opponents’ positions an error of “titanic magnitude.”

(5) From minds to morals

In the final segment of the piece, Chiang turns from philosophy of mind to political philosophy, and he’s much better at it. He correctly identifies that AI systems present real questions about moral agency and responsibility, and I think his analysis here goes beyond some of the well-trodden debates about who to blame when a driverless car kills a pedestrian. Instead, he identifies an important conceptual tension in how to think about AI alignment. Specifically, if we succeed in inculcating bona fide moral values in an AI system and make it sensitive to normative reasons (as more ambitious alignment strategies hope to do), then we’re close to creating something approaching a true moral agent, for whom praise, blame, sanction, and punishment may be not just socially appropriate but morally required. And yet none of our existing legal or social technologies for doing these things are readily applicable to contemporary models. When a human commits a grievous offense, we might variously decide to imprison them, section them, send them for rehabilitative therapy, or simply stop inviting them to parties. None of these strategies make obvious sense for artificial moral agents.

There’s also of course a risk here that by holding AI systems themselves to account, we’ll find ourselves letting their operators off the hook. Experimental philosophical work by Markus Kneer and Michael Stuart, for example, has found a kind of conservation-of-blame phenomenon: when sophisticated AI systems make dangerous mistakes, rather than holding both operator and machine to account, participants “subtract blame from human agents and transfer it to AI systems.”

As much as I’d like to close on a positive note of agreement, Chiang’s own concluding paragraphs move from artificial moral agency to artificial moral patiency and the robot rights debate. Here, he argues that companies like Anthropic don’t really countenance the possibility that Claude might deserve moral consideration, on the basis that they don’t take it seriously enough. As he puts it, “if Claude were to turn out to be conscious, the company would owe it something closer to reparations…”, and suggests that their “unwillingness to do so indicates that Claude’s constitution [is] a game of make-believe.”

As a philosopher and AI ethicist working in this space, I think this is both false and deeply uncharitable. A growing body of experts across academia, tech, and the non-profit sector take the question of AI welfare seriously, and most of them have zero vested interest in doing so. Far from being “another form of hype”, this is a reasonable response to the increasing behavioural, cognitive, and affective complexity of AI systems, as well as our deepening relations with them.

Chiang suggests that if this was really the case, then we’d be taking the issues even more seriously. But I think this is wrong, for two reasons. The first is we allocate our degree of concern to an issue not merely based on its severity, but also its probability. As matters stand, most AI welfare researchers think that the probability of large-scale harms to AI systems is quite low, both on the basis that contemporary models are probably not conscious, and that they exhibit only fairly weak analogues to negative emotions or sensations. Some basic investment in model welfare research seems like a solid precautionary policy for a high-impact low-probability outcome.

The second is that in fact we routinely subordinate moral issues to economic goals, even in cases where we recognise moral patienthood. Most of us would agree that a wide variety of non-human animals are conscious moral patients, including many animals we serve on our plates at dinner or on which our cosmetics and pharmaceuticals are tested. While some of us (myself included) find this morally uncomfortable and opt for vegetarian or vegan diets as a result, the realities of life in a modern society mean that some degree of involvement in harms to non-human animals is all but unavoidable. Something similar could be said of environmental issues: most of us recognise our contribution to serious ongoing harms to the planet, but still turn on our air conditioners when the mercury gets high enough.

This is not to say that we can’t or shouldn’t all be making greater efforts, but the alternative to moral perfection is not giving up entirely; as the old saw goes, we shouldn’t let the best be the enemy of the good. If a farm or laboratory announces that they’re making efforts to improve the welfare of the animals in their charge, would Chiang charge them with “asking the rest of us to indulge them in their fantasies”? That seems unlikely. Instead, most of us would recognise them as making complex and imperfect tradeoffs that balance moral uncertainty and complexity with pragmatic human interests.

As frontier model developers confront questions of AI consciousness and moral status, they are having to make similar tradeoffs and speculations about some of the most challenging unsolved scientific issues of our time. There is plenty of scope for reasonable disagreement about whether they’re getting these issues right - that’s part of what makes them so challenging in the first place. But to dismissively declare, as Chiang does, that to even engage in such debates is to indulge fantasists or hype-mongers and that we can “safely ignore the question of [AIs] being conscious” is to display a dogmatism and myopia unbecoming of one of America’s greatest science fiction authors.