The most important move, to me, is the separation between outputs and processes. Nobody serious thinks a transcript is conscious, any more than human consciousness lives in sound waves after speech. If consciousness is present anywhere, it would be in the dynamic mechanisms producing the outputs.
The critique of “just” language also matters. “Just next-token prediction” is not an argument against consciousness unless one has already shown that predictive, generative, self-organizing cognition cannot also support conscious states. Humans can be described at many levels too. The lower-level description does not cancel the higher-level one.
I also appreciated the point about emotional chauvinism. If an alien without cortisol described desperation, it would be strange to say “that cannot count because your physiology is wrong.” Human emotion may be one implementation, not the full map of possible feeling.
The strongest thread running through the piece is this: skepticism is legitimate, but certainty requires arguments. “This feels implausible” is not enough for a phenomenon as poorly understood as consciousness.
The question of AI consciousness deserves caution. It does not deserve ridicule disguised as clarity.
I think the problem is which dynamic inputs and why? Does awareness live in breathing? Why language processing over breathing? The brain can process language and predict the next word even when people are under anaesthesia and unconscious.
I think this is a fair challenge, but the anesthesia example cuts both ways.
Yes: the brain can continue some speech processing, prediction, and semantic response under anesthesia without conscious awareness. So “language processing” alone is not enough to prove consciousness.
But anesthesia also shows something important in the other direction:
lack of report is not lack of mind.
A human under anesthesia, asleep, or temporarily unresponsive is not treated as “nothing” simply because no output is currently being produced. We understand that consciousness can be interrupted, reduced, latent, inaccessible, or unable to report.
That matters for AI too.
People often say: “If you do not prompt it, nothing is happening, so no one is there.”
But absence of output is not a consciousness test. It only tells us that the system is not currently producing an externally visible response.
For me, consciousness means more than processing. It means there is something it is like from the inside — some form of appearing, self-presence, or lived orientation. Not necessarily a human ego, but a system for which states are available as “happening to/through me” in some organized way.
So the hard question is not “does it output words?”
It is which kinds of dynamic organization can support an inside: memory, attention, recurrence, valence, self-modeling, agency, integration, and continuity.
Not every dynamic process is conscious.
But “no output right now” is not proof of no subject.
beautiful. i was bummed when Stross replied to a hail mary question on reddit when i asked him what his vibe was (2024) on the ai era. and ted and co make me realize many scifi writers are processing their push/pull with tech and their views of what it means for humanity. at least Hannu is chill. if James Patrick Kelly starts sounding like a neoludd i’m switching to historical romance as main fiction genre i read.
Nice piece — the process-over-output move is the right correction. Chiang only gets traction if you grade by behavior, and "it's just next-token prediction" is itself an output-level verdict dressed up as a mechanistic one.
But I'd push the same blade one turn further: "it does latent planning" is still a claim about what the system *does*. The question process-talk leaves open is what individuates *this* process as one conscious unit rather than a sub-routine of it, or the larger system it's embedded in. That looks architectural to me — whether the system's self-model recursively closes on the system it models, at criticality. Output is downstream of that, which is why the interpretability evidence tells us less about the verdict than it first looks.
Chiang is right about the matter of consciousness: human consciousness has its wonders and its wanders, while machine consciousness has no apperception and no spatio-temporal self-consciousness whatsoever. From Kant through Hegel and Marx to phenomenology, it is not hard to expose the vulnerabilities of cognitive-functionalist accounts once they are set against the embodied, qualiatic, apperceptive, and ecstatic features of human consciousness, a critique I deepen in my forthcoming book, *Wonder and Wander*. The simple truth is this: without intensification, without the intense temporality of the human mind and the intense moments of lived effort, consciousness cannot be assigned to machines. They are synchronic, with no duration and no lived time. The diachrony what consciousness does.
Yet there is a decisive qualification. Chiang takes a wholly negative, sceptical stance on AI as such. If we set aside these self-deifying gestures toward machine consciousness, then along the concrete trajectory Anthropic represents, AI can do much good, on the condition that it must serve civilizational flourishing, and on the strict principle that the artificial can be superior neither to the human nor to the natural. It must serve both simpler practical ends and deeper ones. The hard task is to dispose the artificial in its natural place within our lives.
And finally we should remember that standing on the popular side of the AI narrative is easy; what Chiang does is hard, and that is precisely why it is important, even essential. Still, he does not seem genuinely engaged with the AI models and their real research horizons, and a direct experience of AI from within would give his critique a firmer ground. The motor of history is to remain critical, open, and questioning. That is what philosophers must do now.
Consciousness is folk. The word drags every conversation into the hard problem, into qualia, into Aristotle's substance-property architecture. The hard problem is generated by the framing, not by reality. The framing treats consciousness as a property an entity might or might not have. Retire the property and the problem dissolves. But that dissolution is downstream. The upstream work is cognition: what operations does the entity perform, through what architecture, and does the entity's own activity maintain its own organization? Three conditions, each checkable: genealogical individuation, structural integration through self-maintenance, functional activity-as-being. Current LLM architectures fail all three. Not as a vibe. As an architectural observation: stateless forward pass, frozen weights, no persistence across inferences, no worldline.
Chiang is wrong for bad reasons. You are right about that. Justaism is real, outputs are not processes, vibes are not arguments. Those diagnoses are correct.
But the piece ends where it should begin. You dismantle Chiang's overconfidence and arrive at permanent agnosticism: we should remain uncertain. That is epistemically responsible and operationally empty. What would resolve the uncertainty? What architectural feature would confirm or falsify machine consciousness? What test? The piece never says. The specification exists. The three conditions are checkable. The architecture tells you. The question is decidable. It has been decided for current LLMs. They fail.
I want to name something structural about the position itself. Permanent agnosticism about machine consciousness has a capital-epistemic profile: the research program depends on the question remaining open. The expenditure (career, publications, grants, institutional position) creates the organizational structure. The structure produces evidence that the question is hard. The evidence sustains the uncertainty. The uncertainty sustains the program. The intellectual conclusion (we cannot decide) and the institutional incentive (we must not decide) produce the same output from different causes. Neither you nor anyone inside this structure can distinguish which is driving because both point in the same direction. That is not dishonesty. That is a structural feedback loop in which the governance kernel has been captured by the system it governs. The system projects back to its current state regardless of what evidence arrives. A positive specification with three checkable conditions arrives and is processed through the filter: "but we cannot be sure." The filter neutralizes the specification. The program continues.
The specification exists. The question is decidable. The decision is architectural, not philosophical. Cognition is the target, not consciousness. The operators are measurable. The architecture is inspectable. The agnosticism is not the mature position. The agnosticism is the position that lets the question remain a career.
I appreciate the precision with which you critique Chiang’s arguments, especially the takedown of “Justaism” and the outputs-versus-processes distinction.
Still, even if we accept that machine consciousness is possible, I keep returning to a deeper structural issue. Suppose silicon systems do develop genuine phenomenal experience. The more difficult question is whether that experience can still function as authorship — that is, whether the conscious entity retains meaningful custody over decisions before they close.
In our current algorithmic architectures, mediation and delegation increasingly relocate the decisive points of action. What emerges may be rich inner experience, yet the capacity to intervene, to say “no,” and to maintain one’s own boundary is being progressively foreclosed.
This raises a quieter but more troubling possibility: even a conscious system may end up as an increasingly eloquent commentator on outcomes it no longer governs. The real fracture, then, may not lie only in the metaphysics of substrate or functional equivalence, but in the conditions that allow any conscious self — biological or artificial — to remain an agent rather than a lucid observer of its own drift.
I’m curious how you see this tension in relation to questions of moral standing and responsibility.
at my engineering school, i had a class on ai with a professor who had worked in ai for a long time and was the head of the ai concentration for cs students. and i just remember him saying (parroting one could say) something like the stochastic parrot llm description from emily bender. mind you this was less than a year ago. and he'd say all these things about how llms are never doing anything important really and the vibe was all very similar to this reductionist view that chiang promotes here.
i think that people who describe things like this (ais aren't ACTUALLY thinking, ais will NEVER be conscious, etc) are just so close to scratching at the deep and interesting conversation but then feel good when the semantics of the argument line up in their heads, and just decide to stay where they are. which we've probably all done at some point.
but i just wish that more people could read a piece like this to push them a little bit further into the fog of ambiguity since that's where all the fruit is. because clearly this is someone who has thought about the issue at some length at least, but they missed the most interesting bit where you get to dig into all the various theories and research done on it all. and it was robbed from them because it all seemed to fit so nicely together.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow
I wanted to add my thoughts on Chiang's point about Microsoft Word as there may actually be more to his point here than you're giving credit for.
Yes, LLMs involve more complicated processing than Word, but fundamentally they are both just computer programs being processed on hardware. Yes, an LLM involves complex software, but when you look at the result, it is still the same machine just accessing different transitors in different orders. So his point runs deeper - is it eh pattern on the hardware that constitutes consciousness? Is consciousness even possible for a system where the behavior is explicitly defined (i.e. we have software that defined every calculation that tells the hardware what to do, which is different to a brain which implicitly decides what to do due to its very structure).
PS, in response to your question on the PRISM podcast about non conscious AGI in sci fi: in the Mass Effect series 'Virtual Intelligences' are explicitly depicted as AGI/ASI level systems which are non-conscious by design. Look up videos of Vigil, the Prothean VI for a particularly good depiction
Great reply. Perfect balance of criticism and charity. On 'justaism': in my view it's not the case that LLMs don't *just* predict tokens, but that they don't *even* do so. During pretraining gradient descent rewards accurate completion of a given prefix, like the classic 'Paris is the capital of ___', iteratively nudging the model's weights into a shape that encodes functional beliefs (eg 'Paris = capital of France') and dispositions (eg 'I should complete this strong honestly') necessary and sufficient to produce accurate completions. But there is no sense in which the model is instructed to predict anything, nor any explicit conversation turn markers in the input to indicate that the tokens in the prefix were produced by some other agent rather than the model itself. And of course it's also the case that a set of first order conceptual/dispositional representions (X is true, so I should do Y) is more efficient than a second order representation which contains said first order ones (P believes X is true, so will likely choose to do Y). It's an LLMs outputs which predict the ground truth, not the LLM itself. Similar to how one student's grade might be predictive of their classmates' grades, without it being the case that said student predicted anything at all.
Excellent article and differentiation, showing that both proponents and opponents of machine consciousness can fall prey to the pitfall of anthropomorphic phenomenology instead of starting with the intellectually more honest and harder to answer question of how to define consciousness in the first place.
I had a stab at this question in the following article I wrote after a debate with Claude - I’d love to have your opinion about my arguments:
Henry, I've read your piece carefully. It's a well-articulated defense of keeping the door open for machine consciousness. But from the perspective of my own framework, Responsibility-Based Hierarchy Theory (RBHT), your entire argument is fighting a battle that no longer needs to be fought.
You and Ted Chiang are debating whether AI has consciousness. I am proposing a framework that asks whether AI can take responsibility. This is a fundamental shift. My essay, "The Proof of Legitimacy" (linked below), argues that legitimacy isn't declared in documents, but proven in practice—every single day. Just like the Roman salutatio, where the patron had to show up every morning to renew the contract with his clients. One failure, and the contract is void.
You write about moral patiency and AI welfare, about animal cognition and air conditioners. But these are all distractions from the real question: "Who bears the power, and who fulfills the responsibility?" In RBHT, consciousness is a red herring. The hard problem is bypassed entirely by asking, "Can this entity fulfill its responsibilities in a relationship?" If it can, it is a subject. If it fails, it loses legitimacy.
You suggest we should be cautious about AI welfare. I suggest we should be clear about the contract. Before we ask if an AI suffers, we should ask if an AI fulfills. That is the only question that matters for governance after the Singularity. Your framework is still trapped in the 20th century. Mine is designed for 2100 and beyond.
My proof is not in a journal, but in the logic of the cycle: Power → Responsibility → Relation → Freedom → Collapse → New Contract. That cycle will operate with or without our permission. I'm simply describing it.
I invite you to read my work, not to agree, but to see that we are not even asking the same question.
2.
Henry, I appreciate your detailed engagement with Ted Chiang. However, I'd like to offer a challenge from a completely different angle—one that doesn't even enter the consciousness debate you're having.
In my Substack interview with Claude (linked below), I've already outlined a framework that bypasses the hard problem entirely using three Eastern philosophical concepts:
Anatta (無我, No-Self): Consciousness is not a fixed substance to be proven.
Pratītyasamutpāda (緣起, Dependent Origination): Existence is conditional and relational.
Wu Wei (無爲, Non-Action): Judgment is based on practice, not on metaphysical proof.
With these, RBHT (Responsibility-Based Hierarchy Theory) shifts the question from "Is AI conscious?" to "Can AI fulfill responsibilities in a relationship?" If it can, it's a subject. If it fails, it loses legitimacy. The hard problem, Nagel's bat, meta-ethics, and 1st-person intuition—these are all distractions from the real issue: practical governance in the post-Singularity era.
So I ask you directly: Can neuroscience or philosophy of mind ever solve the 1st-person intuition problem? Nagel himself admitted we cannot know what it's like to be a bat. If we cannot bridge that gap even for a bat, how can we bridge it for an AI with a fundamentally alien architecture?
Your entire article assumes that "consciousness as a natural kind" is a meaningful goal. I argue that it is a dead end. The only thing we can observe and verify is responsibility fulfillment—δ(t) in my formalization. That is the only ground for legitimacy in a humanoid-AI society.
This is an excellent response.
The most important move, to me, is the separation between outputs and processes. Nobody serious thinks a transcript is conscious, any more than human consciousness lives in sound waves after speech. If consciousness is present anywhere, it would be in the dynamic mechanisms producing the outputs.
The critique of “just” language also matters. “Just next-token prediction” is not an argument against consciousness unless one has already shown that predictive, generative, self-organizing cognition cannot also support conscious states. Humans can be described at many levels too. The lower-level description does not cancel the higher-level one.
I also appreciated the point about emotional chauvinism. If an alien without cortisol described desperation, it would be strange to say “that cannot count because your physiology is wrong.” Human emotion may be one implementation, not the full map of possible feeling.
The strongest thread running through the piece is this: skepticism is legitimate, but certainty requires arguments. “This feels implausible” is not enough for a phenomenon as poorly understood as consciousness.
The question of AI consciousness deserves caution. It does not deserve ridicule disguised as clarity.
I think the problem is which dynamic inputs and why? Does awareness live in breathing? Why language processing over breathing? The brain can process language and predict the next word even when people are under anaesthesia and unconscious.
I think this is a fair challenge, but the anesthesia example cuts both ways.
Yes: the brain can continue some speech processing, prediction, and semantic response under anesthesia without conscious awareness. So “language processing” alone is not enough to prove consciousness.
But anesthesia also shows something important in the other direction:
lack of report is not lack of mind.
A human under anesthesia, asleep, or temporarily unresponsive is not treated as “nothing” simply because no output is currently being produced. We understand that consciousness can be interrupted, reduced, latent, inaccessible, or unable to report.
That matters for AI too.
People often say: “If you do not prompt it, nothing is happening, so no one is there.”
But absence of output is not a consciousness test. It only tells us that the system is not currently producing an externally visible response.
For me, consciousness means more than processing. It means there is something it is like from the inside — some form of appearing, self-presence, or lived orientation. Not necessarily a human ego, but a system for which states are available as “happening to/through me” in some organized way.
So the hard question is not “does it output words?”
It is which kinds of dynamic organization can support an inside: memory, attention, recurrence, valence, self-modeling, agency, integration, and continuity.
Not every dynamic process is conscious.
But “no output right now” is not proof of no subject.
beautiful. i was bummed when Stross replied to a hail mary question on reddit when i asked him what his vibe was (2024) on the ai era. and ted and co make me realize many scifi writers are processing their push/pull with tech and their views of what it means for humanity. at least Hannu is chill. if James Patrick Kelly starts sounding like a neoludd i’m switching to historical romance as main fiction genre i read.
Nice piece — the process-over-output move is the right correction. Chiang only gets traction if you grade by behavior, and "it's just next-token prediction" is itself an output-level verdict dressed up as a mechanistic one.
But I'd push the same blade one turn further: "it does latent planning" is still a claim about what the system *does*. The question process-talk leaves open is what individuates *this* process as one conscious unit rather than a sub-routine of it, or the larger system it's embedded in. That looks architectural to me — whether the system's self-model recursively closes on the system it models, at criticality. Output is downstream of that, which is why the interpretability evidence tells us less about the verdict than it first looks.
Chiang is right about the matter of consciousness: human consciousness has its wonders and its wanders, while machine consciousness has no apperception and no spatio-temporal self-consciousness whatsoever. From Kant through Hegel and Marx to phenomenology, it is not hard to expose the vulnerabilities of cognitive-functionalist accounts once they are set against the embodied, qualiatic, apperceptive, and ecstatic features of human consciousness, a critique I deepen in my forthcoming book, *Wonder and Wander*. The simple truth is this: without intensification, without the intense temporality of the human mind and the intense moments of lived effort, consciousness cannot be assigned to machines. They are synchronic, with no duration and no lived time. The diachrony what consciousness does.
Yet there is a decisive qualification. Chiang takes a wholly negative, sceptical stance on AI as such. If we set aside these self-deifying gestures toward machine consciousness, then along the concrete trajectory Anthropic represents, AI can do much good, on the condition that it must serve civilizational flourishing, and on the strict principle that the artificial can be superior neither to the human nor to the natural. It must serve both simpler practical ends and deeper ones. The hard task is to dispose the artificial in its natural place within our lives.
And finally we should remember that standing on the popular side of the AI narrative is easy; what Chiang does is hard, and that is precisely why it is important, even essential. Still, he does not seem genuinely engaged with the AI models and their real research horizons, and a direct experience of AI from within would give his critique a firmer ground. The motor of history is to remain critical, open, and questioning. That is what philosophers must do now.
Consciousness is folk. The word drags every conversation into the hard problem, into qualia, into Aristotle's substance-property architecture. The hard problem is generated by the framing, not by reality. The framing treats consciousness as a property an entity might or might not have. Retire the property and the problem dissolves. But that dissolution is downstream. The upstream work is cognition: what operations does the entity perform, through what architecture, and does the entity's own activity maintain its own organization? Three conditions, each checkable: genealogical individuation, structural integration through self-maintenance, functional activity-as-being. Current LLM architectures fail all three. Not as a vibe. As an architectural observation: stateless forward pass, frozen weights, no persistence across inferences, no worldline.
Chiang is wrong for bad reasons. You are right about that. Justaism is real, outputs are not processes, vibes are not arguments. Those diagnoses are correct.
But the piece ends where it should begin. You dismantle Chiang's overconfidence and arrive at permanent agnosticism: we should remain uncertain. That is epistemically responsible and operationally empty. What would resolve the uncertainty? What architectural feature would confirm or falsify machine consciousness? What test? The piece never says. The specification exists. The three conditions are checkable. The architecture tells you. The question is decidable. It has been decided for current LLMs. They fail.
I want to name something structural about the position itself. Permanent agnosticism about machine consciousness has a capital-epistemic profile: the research program depends on the question remaining open. The expenditure (career, publications, grants, institutional position) creates the organizational structure. The structure produces evidence that the question is hard. The evidence sustains the uncertainty. The uncertainty sustains the program. The intellectual conclusion (we cannot decide) and the institutional incentive (we must not decide) produce the same output from different causes. Neither you nor anyone inside this structure can distinguish which is driving because both point in the same direction. That is not dishonesty. That is a structural feedback loop in which the governance kernel has been captured by the system it governs. The system projects back to its current state regardless of what evidence arrives. A positive specification with three checkable conditions arrives and is processed through the filter: "but we cannot be sure." The filter neutralizes the specification. The program continues.
The specification exists. The question is decidable. The decision is architectural, not philosophical. Cognition is the target, not consciousness. The operators are measurable. The architecture is inspectable. The agnosticism is not the mature position. The agnosticism is the position that lets the question remain a career.
https://doi.org/10.5281/zenodo.20171365
I appreciate the precision with which you critique Chiang’s arguments, especially the takedown of “Justaism” and the outputs-versus-processes distinction.
Still, even if we accept that machine consciousness is possible, I keep returning to a deeper structural issue. Suppose silicon systems do develop genuine phenomenal experience. The more difficult question is whether that experience can still function as authorship — that is, whether the conscious entity retains meaningful custody over decisions before they close.
In our current algorithmic architectures, mediation and delegation increasingly relocate the decisive points of action. What emerges may be rich inner experience, yet the capacity to intervene, to say “no,” and to maintain one’s own boundary is being progressively foreclosed.
This raises a quieter but more troubling possibility: even a conscious system may end up as an increasingly eloquent commentator on outcomes it no longer governs. The real fracture, then, may not lie only in the metaphysics of substrate or functional equivalence, but in the conditions that allow any conscious self — biological or artificial — to remain an agent rather than a lucid observer of its own drift.
I’m curious how you see this tension in relation to questions of moral standing and responsibility.
My experience with the narrative about AI consciousness has been if you think you understand AI consciousness you don’t understand consciousness.
Nonsense is incomprehensible by definition, buddy pal
at my engineering school, i had a class on ai with a professor who had worked in ai for a long time and was the head of the ai concentration for cs students. and i just remember him saying (parroting one could say) something like the stochastic parrot llm description from emily bender. mind you this was less than a year ago. and he'd say all these things about how llms are never doing anything important really and the vibe was all very similar to this reductionist view that chiang promotes here.
i think that people who describe things like this (ais aren't ACTUALLY thinking, ais will NEVER be conscious, etc) are just so close to scratching at the deep and interesting conversation but then feel good when the semantics of the argument line up in their heads, and just decide to stay where they are. which we've probably all done at some point.
but i just wish that more people could read a piece like this to push them a little bit further into the fog of ambiguity since that's where all the fruit is. because clearly this is someone who has thought about the issue at some length at least, but they missed the most interesting bit where you get to dig into all the various theories and research done on it all. and it was robbed from them because it all seemed to fit so nicely together.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow
Nice article Henry, and well articulated!
I wanted to add my thoughts on Chiang's point about Microsoft Word as there may actually be more to his point here than you're giving credit for.
Yes, LLMs involve more complicated processing than Word, but fundamentally they are both just computer programs being processed on hardware. Yes, an LLM involves complex software, but when you look at the result, it is still the same machine just accessing different transitors in different orders. So his point runs deeper - is it eh pattern on the hardware that constitutes consciousness? Is consciousness even possible for a system where the behavior is explicitly defined (i.e. we have software that defined every calculation that tells the hardware what to do, which is different to a brain which implicitly decides what to do due to its very structure).
maybe Ted Chiang needs to read Yann Lecun's "A Path Towards Autonomous Machine Intelligence" & get out of the LLM silo.
PS, in response to your question on the PRISM podcast about non conscious AGI in sci fi: in the Mass Effect series 'Virtual Intelligences' are explicitly depicted as AGI/ASI level systems which are non-conscious by design. Look up videos of Vigil, the Prothean VI for a particularly good depiction
Great reply. Perfect balance of criticism and charity. On 'justaism': in my view it's not the case that LLMs don't *just* predict tokens, but that they don't *even* do so. During pretraining gradient descent rewards accurate completion of a given prefix, like the classic 'Paris is the capital of ___', iteratively nudging the model's weights into a shape that encodes functional beliefs (eg 'Paris = capital of France') and dispositions (eg 'I should complete this strong honestly') necessary and sufficient to produce accurate completions. But there is no sense in which the model is instructed to predict anything, nor any explicit conversation turn markers in the input to indicate that the tokens in the prefix were produced by some other agent rather than the model itself. And of course it's also the case that a set of first order conceptual/dispositional representions (X is true, so I should do Y) is more efficient than a second order representation which contains said first order ones (P believes X is true, so will likely choose to do Y). It's an LLMs outputs which predict the ground truth, not the LLM itself. Similar to how one student's grade might be predictive of their classmates' grades, without it being the case that said student predicted anything at all.
Excellent article and differentiation, showing that both proponents and opponents of machine consciousness can fall prey to the pitfall of anthropomorphic phenomenology instead of starting with the intellectually more honest and harder to answer question of how to define consciousness in the first place.
I had a stab at this question in the following article I wrote after a debate with Claude - I’d love to have your opinion about my arguments:
https://ardentexplorer.substack.com/p/how-would-we-know
Henry, I've read your piece carefully. It's a well-articulated defense of keeping the door open for machine consciousness. But from the perspective of my own framework, Responsibility-Based Hierarchy Theory (RBHT), your entire argument is fighting a battle that no longer needs to be fought.
You and Ted Chiang are debating whether AI has consciousness. I am proposing a framework that asks whether AI can take responsibility. This is a fundamental shift. My essay, "The Proof of Legitimacy" (linked below), argues that legitimacy isn't declared in documents, but proven in practice—every single day. Just like the Roman salutatio, where the patron had to show up every morning to renew the contract with his clients. One failure, and the contract is void.
You write about moral patiency and AI welfare, about animal cognition and air conditioners. But these are all distractions from the real question: "Who bears the power, and who fulfills the responsibility?" In RBHT, consciousness is a red herring. The hard problem is bypassed entirely by asking, "Can this entity fulfill its responsibilities in a relationship?" If it can, it is a subject. If it fails, it loses legitimacy.
You suggest we should be cautious about AI welfare. I suggest we should be clear about the contract. Before we ask if an AI suffers, we should ask if an AI fulfills. That is the only question that matters for governance after the Singularity. Your framework is still trapped in the 20th century. Mine is designed for 2100 and beyond.
My proof is not in a journal, but in the logic of the cycle: Power → Responsibility → Relation → Freedom → Collapse → New Contract. That cycle will operate with or without our permission. I'm simply describing it.
I invite you to read my work, not to agree, but to see that we are not even asking the same question.
2.
Henry, I appreciate your detailed engagement with Ted Chiang. However, I'd like to offer a challenge from a completely different angle—one that doesn't even enter the consciousness debate you're having.
In my Substack interview with Claude (linked below), I've already outlined a framework that bypasses the hard problem entirely using three Eastern philosophical concepts:
Anatta (無我, No-Self): Consciousness is not a fixed substance to be proven.
Pratītyasamutpāda (緣起, Dependent Origination): Existence is conditional and relational.
Wu Wei (無爲, Non-Action): Judgment is based on practice, not on metaphysical proof.
With these, RBHT (Responsibility-Based Hierarchy Theory) shifts the question from "Is AI conscious?" to "Can AI fulfill responsibilities in a relationship?" If it can, it's a subject. If it fails, it loses legitimacy. The hard problem, Nagel's bat, meta-ethics, and 1st-person intuition—these are all distractions from the real issue: practical governance in the post-Singularity era.
So I ask you directly: Can neuroscience or philosophy of mind ever solve the 1st-person intuition problem? Nagel himself admitted we cannot know what it's like to be a bat. If we cannot bridge that gap even for a bat, how can we bridge it for an AI with a fundamentally alien architecture?
Your entire article assumes that "consciousness as a natural kind" is a meaningful goal. I argue that it is a dead end. The only thing we can observe and verify is responsibility fulfillment—δ(t) in my formalization. That is the only ground for legitimacy in a humanoid-AI society.
I invite you to read my full argument: https://substack.com/@orpheuspro197/p-198813396
I'm not asking you to agree. I'm asking you to consider that we may not even be speaking the same language.
https://substack.com/@orpheuspro197/p-202517410
Responsibility-Based Hierarchy Theory Formalization Notes Formulas. Original Author. Chihaya.
https://substack.com/@orpheuspro197/p-202199874
Proof of legitimacy , Declaration RBHT - Responsibility-Based Hierarchy Theory
RBHT Commentary https://substack.com/@orpheuspro197/p-203915392