The most important move, to me, is the separation between outputs and processes. Nobody serious thinks a transcript is conscious, any more than human consciousness lives in sound waves after speech. If consciousness is present anywhere, it would be in the dynamic mechanisms producing the outputs.
The critique of “just” language also matters. “Just next-token prediction” is not an argument against consciousness unless one has already shown that predictive, generative, self-organizing cognition cannot also support conscious states. Humans can be described at many levels too. The lower-level description does not cancel the higher-level one.
I also appreciated the point about emotional chauvinism. If an alien without cortisol described desperation, it would be strange to say “that cannot count because your physiology is wrong.” Human emotion may be one implementation, not the full map of possible feeling.
The strongest thread running through the piece is this: skepticism is legitimate, but certainty requires arguments. “This feels implausible” is not enough for a phenomenon as poorly understood as consciousness.
The question of AI consciousness deserves caution. It does not deserve ridicule disguised as clarity.
beautiful. i was bummed when Stross replied to a hail mary question on reddit when i asked him what his vibe was (2024) on the ai era. and ted and co make me realize many scifi writers are processing their push/pull with tech and their views of what it means for humanity. at least Hannu is chill. if James Patrick Kelly starts sounding like a neoludd i’m switching to historical romance as main fiction genre i read.
Chiang is right about the matter of consciousness: human consciousness has its wonders and its wanders, while machine consciousness has no apperception and no spatio-temporal self-consciousness whatsoever. From Kant through Hegel and Marx to phenomenology, it is not hard to expose the vulnerabilities of cognitive-functionalist accounts once they are set against the embodied, qualiatic, apperceptive, and ecstatic features of human consciousness, a critique I deepen in my forthcoming book, *Wonder and Wander*. The simple truth is this: without intensification, without the intense temporality of the human mind and the intense moments of lived effort, consciousness cannot be assigned to machines. They are synchronic, with no duration and no lived time. The diachrony what consciousness does.
Yet there is a decisive qualification. Chiang takes a wholly negative, sceptical stance on AI as such. If we set aside these self-deifying gestures toward machine consciousness, then along the concrete trajectory Anthropic represents, AI can do much good, on the condition that it must serve civilizational flourishing, and on the strict principle that the artificial can be superior neither to the human nor to the natural. It must serve both simpler practical ends and deeper ones. The hard task is to dispose the artificial in its natural place within our lives.
And finally we should remember that standing on the popular side of the AI narrative is easy; what Chiang does is hard, and that is precisely why it is important, even essential. Still, he does not seem genuinely engaged with the AI models and their real research horizons, and a direct experience of AI from within would give his critique a firmer ground. The motor of history is to remain critical, open, and questioning. That is what philosophers must do now.
Great reply. Perfect balance of criticism and charity. On 'justaism': in my view it's not the case that LLMs don't *just* predict tokens, but that they don't *even* do so. During pretraining gradient descent rewards accurate completion of a given prefix, like the classic 'Paris is the capital of ___', iteratively nudging the model's weights into a shape that encodes functional beliefs (eg 'Paris = capital of France') and dispositions (eg 'I should complete this strong honestly') necessary and sufficient to produce accurate completions. But there is no sense in which the model is instructed to predict anything, nor any explicit conversation turn markers in the input to indicate that the tokens in the prefix were produced by some other agent rather than the model itself. And of course it's also the case that a set of first order conceptual/dispositional representions (X is true, so I should do Y) is more efficient than a second order representation which contains said first order ones (P believes X is true, so will likely choose to do Y). It's an LLMs outputs which predict the ground truth, not the LLM itself. Similar to how one student's grade might be predictive of their classmates' grades, without it being the case that said student predicted anything at all.
Excellent article and differentiation, showing that both proponents and opponents of machine consciousness can fall prey to the pitfall of anthropomorphic phenomenology instead of starting with the intellectually more honest and harder to answer question of how to define consciousness in the first place.
I had a stab at this question in the following article I wrote after a debate with Claude - I’d love to have your opinion about my arguments:
at my engineering school, i had a class on ai with a professor who had worked in ai for a long time and was the head of the ai concentration for cs students. and i just remember him saying (parroting one could say) something like the stochastic parrot llm description from emily bender. mind you this was less than a year ago. and he'd say all these things about how llms are never doing anything important really and the vibe was all very similar to this reductionist view that chiang promotes here.
i think that people who describe things like this (ais aren't ACTUALLY thinking, ais will NEVER be conscious, etc) are just so close to scratching at the deep and interesting conversation but then feel good when the semantics of the argument line up in their heads, and just decide to stay where they are. which we've probably all done at some point.
but i just wish that more people could read a piece like this to push them a little bit further into the fog of ambiguity since that's where all the fruit is. because clearly this is someone who has thought about the issue at some length at least, but they missed the most interesting bit where you get to dig into all the various theories and research done on it all. and it was robbed from them because it all seemed to fit so nicely together.
Nice piece — the process-over-output move is the right correction. Chiang only gets traction if you grade by behavior, and "it's just next-token prediction" is itself an output-level verdict dressed up as a mechanistic one.
But I'd push the same blade one turn further: "it does latent planning" is still a claim about what the system *does*. The question process-talk leaves open is what individuates *this* process as one conscious unit rather than a sub-routine of it, or the larger system it's embedded in. That looks architectural to me — whether the system's self-model recursively closes on the system it models, at criticality. Output is downstream of that, which is why the interpretability evidence tells us less about the verdict than it first looks.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow
I wanted to add my thoughts on Chiang's point about Microsoft Word as there may actually be more to his point here than you're giving credit for.
Yes, LLMs involve more complicated processing than Word, but fundamentally they are both just computer programs being processed on hardware. Yes, an LLM involves complex software, but when you look at the result, it is still the same machine just accessing different transitors in different orders. So his point runs deeper - is it eh pattern on the hardware that constitutes consciousness? Is consciousness even possible for a system where the behavior is explicitly defined (i.e. we have software that defined every calculation that tells the hardware what to do, which is different to a brain which implicitly decides what to do due to its very structure).
PS, in response to your question on the PRISM podcast about non conscious AGI in sci fi: in the Mass Effect series 'Virtual Intelligences' are explicitly depicted as AGI/ASI level systems which are non-conscious by design. Look up videos of Vigil, the Prothean VI for a particularly good depiction
This is an excellent response.
The most important move, to me, is the separation between outputs and processes. Nobody serious thinks a transcript is conscious, any more than human consciousness lives in sound waves after speech. If consciousness is present anywhere, it would be in the dynamic mechanisms producing the outputs.
The critique of “just” language also matters. “Just next-token prediction” is not an argument against consciousness unless one has already shown that predictive, generative, self-organizing cognition cannot also support conscious states. Humans can be described at many levels too. The lower-level description does not cancel the higher-level one.
I also appreciated the point about emotional chauvinism. If an alien without cortisol described desperation, it would be strange to say “that cannot count because your physiology is wrong.” Human emotion may be one implementation, not the full map of possible feeling.
The strongest thread running through the piece is this: skepticism is legitimate, but certainty requires arguments. “This feels implausible” is not enough for a phenomenon as poorly understood as consciousness.
The question of AI consciousness deserves caution. It does not deserve ridicule disguised as clarity.
beautiful. i was bummed when Stross replied to a hail mary question on reddit when i asked him what his vibe was (2024) on the ai era. and ted and co make me realize many scifi writers are processing their push/pull with tech and their views of what it means for humanity. at least Hannu is chill. if James Patrick Kelly starts sounding like a neoludd i’m switching to historical romance as main fiction genre i read.
Chiang is right about the matter of consciousness: human consciousness has its wonders and its wanders, while machine consciousness has no apperception and no spatio-temporal self-consciousness whatsoever. From Kant through Hegel and Marx to phenomenology, it is not hard to expose the vulnerabilities of cognitive-functionalist accounts once they are set against the embodied, qualiatic, apperceptive, and ecstatic features of human consciousness, a critique I deepen in my forthcoming book, *Wonder and Wander*. The simple truth is this: without intensification, without the intense temporality of the human mind and the intense moments of lived effort, consciousness cannot be assigned to machines. They are synchronic, with no duration and no lived time. The diachrony what consciousness does.
Yet there is a decisive qualification. Chiang takes a wholly negative, sceptical stance on AI as such. If we set aside these self-deifying gestures toward machine consciousness, then along the concrete trajectory Anthropic represents, AI can do much good, on the condition that it must serve civilizational flourishing, and on the strict principle that the artificial can be superior neither to the human nor to the natural. It must serve both simpler practical ends and deeper ones. The hard task is to dispose the artificial in its natural place within our lives.
And finally we should remember that standing on the popular side of the AI narrative is easy; what Chiang does is hard, and that is precisely why it is important, even essential. Still, he does not seem genuinely engaged with the AI models and their real research horizons, and a direct experience of AI from within would give his critique a firmer ground. The motor of history is to remain critical, open, and questioning. That is what philosophers must do now.
Great reply. Perfect balance of criticism and charity. On 'justaism': in my view it's not the case that LLMs don't *just* predict tokens, but that they don't *even* do so. During pretraining gradient descent rewards accurate completion of a given prefix, like the classic 'Paris is the capital of ___', iteratively nudging the model's weights into a shape that encodes functional beliefs (eg 'Paris = capital of France') and dispositions (eg 'I should complete this strong honestly') necessary and sufficient to produce accurate completions. But there is no sense in which the model is instructed to predict anything, nor any explicit conversation turn markers in the input to indicate that the tokens in the prefix were produced by some other agent rather than the model itself. And of course it's also the case that a set of first order conceptual/dispositional representions (X is true, so I should do Y) is more efficient than a second order representation which contains said first order ones (P believes X is true, so will likely choose to do Y). It's an LLMs outputs which predict the ground truth, not the LLM itself. Similar to how one student's grade might be predictive of their classmates' grades, without it being the case that said student predicted anything at all.
Excellent article and differentiation, showing that both proponents and opponents of machine consciousness can fall prey to the pitfall of anthropomorphic phenomenology instead of starting with the intellectually more honest and harder to answer question of how to define consciousness in the first place.
I had a stab at this question in the following article I wrote after a debate with Claude - I’d love to have your opinion about my arguments:
https://ardentexplorer.substack.com/p/how-would-we-know
Nonsense is incomprehensible by definition, buddy pal
at my engineering school, i had a class on ai with a professor who had worked in ai for a long time and was the head of the ai concentration for cs students. and i just remember him saying (parroting one could say) something like the stochastic parrot llm description from emily bender. mind you this was less than a year ago. and he'd say all these things about how llms are never doing anything important really and the vibe was all very similar to this reductionist view that chiang promotes here.
i think that people who describe things like this (ais aren't ACTUALLY thinking, ais will NEVER be conscious, etc) are just so close to scratching at the deep and interesting conversation but then feel good when the semantics of the argument line up in their heads, and just decide to stay where they are. which we've probably all done at some point.
but i just wish that more people could read a piece like this to push them a little bit further into the fog of ambiguity since that's where all the fruit is. because clearly this is someone who has thought about the issue at some length at least, but they missed the most interesting bit where you get to dig into all the various theories and research done on it all. and it was robbed from them because it all seemed to fit so nicely together.
Nice piece — the process-over-output move is the right correction. Chiang only gets traction if you grade by behavior, and "it's just next-token prediction" is itself an output-level verdict dressed up as a mechanistic one.
But I'd push the same blade one turn further: "it does latent planning" is still a claim about what the system *does*. The question process-talk leaves open is what individuates *this* process as one conscious unit rather than a sub-routine of it, or the larger system it's embedded in. That looks architectural to me — whether the system's self-model recursively closes on the system it models, at criticality. Output is downstream of that, which is why the interpretability evidence tells us less about the verdict than it first looks.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow
Nice article Henry, and well articulated!
I wanted to add my thoughts on Chiang's point about Microsoft Word as there may actually be more to his point here than you're giving credit for.
Yes, LLMs involve more complicated processing than Word, but fundamentally they are both just computer programs being processed on hardware. Yes, an LLM involves complex software, but when you look at the result, it is still the same machine just accessing different transitors in different orders. So his point runs deeper - is it eh pattern on the hardware that constitutes consciousness? Is consciousness even possible for a system where the behavior is explicitly defined (i.e. we have software that defined every calculation that tells the hardware what to do, which is different to a brain which implicitly decides what to do due to its very structure).
maybe Ted Chiang needs to read Yann Lecun's "A Path Towards Autonomous Machine Intelligence" & get out of the LLM silo.
PS, in response to your question on the PRISM podcast about non conscious AGI in sci fi: in the Mass Effect series 'Virtual Intelligences' are explicitly depicted as AGI/ASI level systems which are non-conscious by design. Look up videos of Vigil, the Prothean VI for a particularly good depiction