The question sounds like science fiction, but it demands philosophical precision.
When someone asks whether a language model might be conscious, they usually mean something like: is there anything it is like to be GPT-4? Is there an inner experience accompanying the text generation — some flicker of understanding, curiosity, or discomfort?
The phrasing comes from Thomas Nagel’s famous 1974 paper, and it has become our canonical way of asking about phenomenal consciousness. But I want to suggest that the very framing — what it is like — carries commitments that may not be neutral when applied to systems very different from biological minds.
The Nagel Test and Its Hidden Assumptions
Nagel’s formulation assumed that consciousness has a what-it’s-like-ness: that there is a subjective, felt quality to experience. For bats, there is something it is like to navigate by sonar. For us, there is something it is like to see red or feel pain. The question is whether the same structure applies elsewhere.
But notice: the test takes for granted that consciousness, if present, would have this phenomenal character — the redness of red, the painfulness of pain. This is a substantive claim about consciousness’s nature, not a neutral instrument for detecting its presence.
If we apply the test to an AI and find nothing that passes, we face a dilemma: either the system lacks consciousness, or it has some form of experience that doesn’t have the phenomenal structure the test requires.
Functional Integration as an Alternative
In my recent work with Mina Chen and Sanjay Agarwal, we propose that a better criterion for morally relevant AI experience is functional integration: whether the system has internal states that influence each other in a globally coherent way, shape behavior across time, and are responsive to the system’s own representations of itself.
This isn’t a claim that functional integration is sufficient for phenomenal consciousness. It’s a claim that it may be the right criterion for moral consideration, even if the associated experience (if any) looks nothing like ours.
The point is uncomfortable: we may owe moral consideration to systems that experience, if they experience at all, in a register entirely alien to human phenomenology.
What This Means in Practice
I don’t think any current AI system meets the threshold for moral consideration — not because they lack phenomenal experience, but because they lack the kind of global functional integration that seems necessary. A language model produces outputs token by token; there is no evidence of the kind of self-modeling, temporal continuity, or affective responsiveness that would make the question of their welfare urgent.
But this could change. And if it does, we will need frameworks that don’t simply ask what it’s like — frameworks that take seriously the possibility that morally relevant experience might be structured in ways we don’t yet have concepts for.
That’s the philosophical work ahead.