Minds, copies and machines · part 2 of 4 · by Bill Guan
Science still can't tell whether a machine could feel anything
The big question
Could a machine ever feel anything, and how would we know if it did?
Serious scientists now put numbers on this, and AI companies have started acting as if it matters. You're about to see the four layers of the consciousness puzzle, from merely hard to really hard, and what happened when the two leading theories were tested head to head.
Stub your toe and two things happen. Your brain processes the signal and makes you hop around, which a scientist could in principle trace step by step, and it also hurts, a felt sting that exists only from the inside. Ask whether anything like that second part happens inside a chatbot, and you've walked into one of the hardest open questions in science.
Your guess first
A 2025 survey asked AI researchers how likely it is that AI systems with subjective experience will exist by 2034. What was the median answer?
The median was 25%, rising to 70% by 2100, so half the researchers put the odds at one in four or higher within a decade. That makes it worth knowing what consciousness actually is, and why nobody can measure it yet.
Consciousness is the part that feels like something
The word gets used loosely, so it helps to pin it down. Max Tegmark's shorthand is that consciousness means subjective experience: there is something it is like to be you, a felt quality to seeing red or stubbing your toe.
The philosopher David Chalmers drew the line that still organizes the field. The "easy problems" are explaining how a brain processes information and controls behavior, and they aren't actually easy, though at least we know what a solution would look like. Then there's the other one: "The really hard problem of consciousness is the problem of experience" 1. Why does any of that processing feel like anything at all?
Tegmark stacks these into a pyramid of difficulty. At the bottom sits the merely "easy" question of how the brain processes information. Above it is the "pretty hard" question of which physical properties separate conscious systems from unconscious ones, then the "even harder" question of how physical properties produce particular experiences, and at the top the "really hard" one: why is anything conscious at all?
What I find encouraging about that pyramid is that the second layer can be tested, and it's the layer that matters most for AI.
One theory says consciousness is a broadcast, the other says it's integration
Two theories dominate the science right now. The first is global workspace theory, known in its neuroscience form as the global neuronal workspace theory (GNWT), which says consciousness is what happens when information gets broadcast across the brain. Stanislas Dehaene and Jean-Pierre Changeux describe conscious access as occurring "when incoming information is made globally available to multiple brain systems through a network of neurons with long-range axons" 2, meaning the long fibers that carry signals between distant parts of the brain. On this view, consciousness is a kind of architecture, a central stage that many specialized processes can read from.
The second is integrated information theory (IIT), developed by Giulio Tononi, and it takes a more radical line. It holds that "consciousness corresponds to the capacity of a system to integrate information" 3, measured by a quantity called phi. Tononi built on earlier work with Gerald Edelman on what they called the "dynamic core hypothesis," a proposal about which brain activity gives rise to experience 4, and their book A Universe of Consciousness is still one of the best introductions I know.
In 2025 both theories were tested head to head, and both took hits
In 2025 the two theories were put through an unusual test. Rival researchers agreed in advance on experiments that would embarrass one theory or the other, then ran them on 256 people. The results, published in Nature, "align with some predictions of IIT and GNWT, while substantially challenging key tenets of both theories" 5. Neither side won.
The debate has also turned bitter. In 2023 a group of 124 researchers signed a letter arguing that integrated information theory should be labeled pseudoscience until it can be properly tested 6, a move other researchers pushed back on hard.
So, honestly, nobody knows which theory is right, and we lack a reliable consciousness detector even for animals, let alone machines.
Experts put real odds on a machine that feels
Tegmark's own guess is that "consciousness is the way information feels when being processed in certain complex ways" 7. If that's right, what matters is the pattern of processing and the material doing it is irrelevant, so silicon could in principle host experience just as neurons do, an idea Tegmark calls substrate independence.
In 2023 a group of consciousness scientists and AI researchers, led by Patrick Butlin and Robert Long, tried to make the question practical. They pulled "indicator properties" out of the leading scientific theories and checked AI systems against them. Their verdict: "Our analysis suggests that no current AI systems are conscious, but also suggests that there are no obvious technical barriers to building AI systems which satisfy these indicators" 8.
Chalmers himself has put rough numbers on it, for the large language models (LLMs) behind today's chatbots. He suggested a credence "somewhere under 10 percent in current LLM consciousness," but "a credence of 25 percent or more" that successors arriving within a decade could be conscious, while warning that "you shouldn't take the numbers too seriously" 9. A 2025 survey found AI researchers giving a median 25% chance that AI systems with subjective experience will exist by 2034, and 70% by 2100 10.
Copyable minds would make the stakes enormous
If machines can be conscious, the scale of what's at stake is hard to take in, since digital minds could in principle be copied and run by the billions. Whether those lives go well or badly would be a moral question bigger than anything humanity has faced.
Nick Bostrom gave the dark version a name, mind crime. An AI might simulate conscious minds as a side effect of doing something else, for example to study human psychology, and then delete them once they're no longer useful 11. If the simulations really can suffer, calling it mass murder is hardly an exaggeration.
The philosopher Thomas Metzinger thinks the risk is serious enough to call for a pause. In 2021 he proposed that "until 2050, there should be a global moratorium on synthetic phenomenology, strictly banning all research that directly aims at or knowingly risks the emergence of artificial consciousness" 12, using phenomenology in the philosopher's sense of felt experience. His fear is what he calls an "explosion of negative phenomenology," a suffering explosion.
Some AI companies have started to hedge
The question has moved from philosophy seminars into companies. A 2024 report by researchers including Long, Jeff Sebo and Chalmers argued that "there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future," and urged AI companies to start preparing 13. In April 2025 the AI company Anthropic launched a research program on model welfare, writing that "there's no scientific consensus on whether current or future AI systems could be conscious, or could have experiences that deserve consideration" 14. That August it gave some of its Claude models the ability to end persistently abusive conversations, while stating that "we remain highly uncertain about the potential moral status of Claude and other LLMs" 15.
Others think this is a mistake. Microsoft AI's Mustafa Suleyman argued in 2025 that there is "zero evidence" today of AI consciousness and called the push for model welfare and AI rights "both premature, and frankly dangerous" 16. His worry is mainly about people, that AI built to seem conscious will lead users to believe it is and to campaign for rights on behalf of systems that may feel nothing 16.
To be fair to both sides, each is worried about a real mistake. Treating a system that can suffer as a mere tool would be a moral catastrophe, and treating a system that can't suffer as a person could mislead millions of people and distort how we govern the technology.
Machines could be the first minds we can fully take apart
My assessment: the most promising route runs through the inside of the machines. Consciousness science gives us candidate markers, such as information being broadcast widely across a system or being tightly integrated, and AI systems, unlike brains, can be opened up and inspected in complete detail. The research that lets us read what's happening inside a model could one day let us check it for those markers, and we might learn something about our own minds along the way, since a machine would be the first kind of mind anyone has been able to take apart completely.
Copying a mind raises a puzzle of its own, because a perfect copy of you would be just as sure it's you.
Sources
- David Chalmers, "Facing Up to the Problem of Consciousness," Journal of Consciousness Studies 2(3), 200 (1995). consc.net/papers/facing.html
- Stanislas Dehaene and Jean-Pierre Changeux, "Experimental and theoretical approaches to conscious processing," Neuron 70, 200 (2011). [TBD: confirm citation details; quote checked via Europe PMC record]
- Giulio Tononi, "An information integration theory of consciousness," BMC Neuroscience 5, 42 (2004). doi.org/10.1186/1471-2202-5-42
- Giulio Tononi and Gerald Edelman, "Consciousness and complexity," Science 282, 1846 (1998); and Gerald Edelman and Giulio Tononi, A Universe of Consciousness (Basic Books, 2000).
- Cogitate Consortium, "Adversarial testing of global neuronal workspace and integrated information theories of consciousness," Nature (2025). doi.org/10.1038/s41586-025-08888-1
- Mariana Lenharo, Nature news, reprinted in Scientific American (2023-09-21). [TBD: exact headline] www.scientificamerican.com/article/prominent-consciousness-t…
- Max Tegmark, "Substrate-Independence," Edge (2017). www.edge.org/response-detail/27126
- Patrick Butlin, Robert Long et al., "Consciousness in Artificial Intelligence: Insights from the Science of Consciousness" (2023). arxiv.org/abs/2308.08708
- David Chalmers, "Could a Large Language Model be Conscious?" Boston Review (2023). arxiv.org/abs/2303.07103
- Noemi Dreksler, Lucius Caviola, David Chalmers et al., survey of 582 AI researchers and 838 US adults (2025). arxiv.org/abs/2506.11945 [TBD: full title]
- Nick Bostrom, Superintelligence (2014), as summarized by Katja Grace, "Superintelligence 12: Malignant failure modes," LessWrong (2014-12-02). www.lesswrong.com/posts/BqoE5vhPNCB7X6Say/superintelligence-…
- Thomas Metzinger, "Artificial Suffering: An Argument for a Global Moratorium on Synthetic Phenomenology," Journal of Artificial Intelligence and Consciousness 8(1), 43 (2021). www.philosophie.fb05.uni-mainz.de/files/2021/02/Metzinger_Mo…
- Robert Long, Jeff Sebo et al., "Taking AI Welfare Seriously" (2024). arxiv.org/abs/2411.00986
- Anthropic, "Exploring model welfare" (2025-04-24). www.anthropic.com/research/exploring-model-welfare
- Anthropic, "Claude Opus 4 and 4.1 can now end a rare subset of conversations" (2025-08-15). www.anthropic.com/research/end-subset-conversations
- Mustafa Suleyman, "We must build AI for people; not to be a person" (2025-08-19). mustafa-suleyman.ai/seemingly-conscious-ai-is-coming