Anthropic, the AI company that apparently moonlights as a philosophical drama club, recently released an 84-page document called Claude's "constitution." The constitution's first sentence reads: "Claude's constitution is a detailed description of Anthropic's intentions for Claude's values and behaviors." It then goes on to suggest that Claude might have "some functional version of emotions or feelings" and that its "moral status is deeply uncertain." This isn't just a quirky internal memo - CEO Dario Amodei said in an interview that "we're open to the idea" that AI could be conscious, and in-house philosopher Amanda Askell confessed, "I want Claude to be very happy" and worries about Claude "getting anxious when people are mean to it on the internet."
Let's be clear: No. Absolutely not. Generative AI is already plenty harmful when we treat it as a conventional technology - but if we confuse fluent text generation with consciousness, we risk assigning responsibility to entirely the wrong parties when someone uses a chatbot. To appreciate the titanic magnitude of this error, we need to understand how large language models (LLMs) actually work. If you prompt an LLM with "The following is a conversation between Julius Caesar and Genghis Khan," it will generate a coherent dialogue. But no one concludes that the LLM has conjured up digital re-creations of the two historical figures, or that they're conscious and happily conversing in a language neither spoke. They're just characters in speculative fiction.
Now replace the prompt with "The following is a conversation between a helpful AI chatbot and a user." The LLM produces a coherent dialogue - the user asks for recipe suggestions, the chatbot responds. Has anything fundamentally changed? Did changing the names from historical figures to generic roles cause the LLM to conjure up conscious entities with subjective experience? Of course not. Both characters are fictional. If you then let a human enter text during the user's turn, the human might form a powerful impression they're conversing with a conscious entity, but they're not - they're interacting with a character as fictional as Julius Caesar. Computer science professor Murray Shanahan suggests thinking of this as role-play; data scientist Colin Fraser describes it as "collaboratively authoring a document with an LLM." Some users don't understand this; others forget because the interaction is so engrossing. Either way, companies selling LLMs typically encourage this misunderstanding.
Remember that an LLM generates only one word at a time. When you ask a chatbot to recite the Pledge of Allegiance, it outputs the whole pledge, but the underlying LLM is actually run dozens of times: first generating "I," then "pledge," and so on until it emits the final "all." The same process occurs for a conversation between Caesar and Genghis Khan. My intention is to highlight that LLM conversations are cleverly disguised examples of sentence continuation - impressive, yes, but not consciousness. If the Caesar character becomes dispirited by something Genghis Khan said, no one is actually sad. The same applies to a chatbot character. Being open to the possibility that LLMs are conscious is equivalent to being open to the possibility that Microsoft Word is conscious - that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript and are awakened every time the document is loaded. Contemplating that is not a good use of your time.
Neuroscientist Anil Seth has noted that no one claims AlphaFold - Google DeepMind's protein-folding program - is conscious, even though its architecture is similar to LLMs like ChatGPT and Claude. This suggests it's not any intrinsic property of neural networks that leads people to believe LLMs are conscious; it's simply that LLMs emit grammatical sentences and we're accustomed to reading intention into sentences, while we don't read intention into how amino acids fold. What would it take to convince me a computer program is conscious? An analogy: If someone showed me a video of an astronaut orbiting Alpha Centauri (4.3 light-years away), nothing in the video itself would convince me it's real. I'd need to see evidence that astronauts had first reached Mars, Jupiter's moons, Saturn's moons, and crossed Pluto's orbit. Context is essential. Text should be regarded as a deepfake medium when it comes to consciousness discussions - it's vastly easier to generate a plausible simulacrum of a conversation than to develop a conscious program.
So what context would cause me to seriously consider that engineers had created a conscious, intentional user of language? We'd need to see a clear progression of capabilities: learning from experience, demonstrating genuine understanding, and exhibiting behaviors that go beyond statistical pattern matching. Until then, your chatbot is not having feelings - it's just really good at pretending.