Remember when the biggest threat to academic integrity was a student copying from Wikipedia? Those were simpler times. Now we have Pangram, an AI-detection tool that has become the de facto judge, jury, and executioner for suspected bot-written text - even though it might be making things worse.
Pangram has been at the center of nearly every high-profile AI-writing accusation lately. It flagged a horror novel pulled days before its release, suggested chatbots wrote articles in The New York Times, raised eyebrows over award-winning short stories, and even implicated significant chunks of Pope Leo XIV's encyclical about AI dangers. Universities use it to vet student work; scientific associations scan research papers with it. When panic over AI writing strikes, Pangram is the go-to panic button.
Just a few years ago, reliable AI detection seemed impossible. In 2023, ZeroGPT declared the U.S. Constitution AI-written, and OpenAI abandoned its own detector due to a 'low rate of accuracy.' That was back when ChatGPT's writing was noticeably worse. Now, detection tools have improved dramatically - and Pangram has emerged as the gold standard. Paste text in, and it tells you what's 'AI Generated,' 'AI Assisted,' or 'Human Written.'
But here's the thing about gold standards: they can still tarnish. Pangram's CEO Max Spero claims the algorithm incorrectly flags human text as AI only about once in every 10,000 times. 'There is a great responsibility, a huge weight in saying something is AI-generated,' Spero told me. 'The only reason we do so is because we're extremely confident.' Independent analyses back him up - one University of Chicago paper found almost no false positives on some 3,000 sample texts.
However, Pangram's ability to guarantee something was written by a human is shakier. The false-negative rate - how often it incorrectly labels AI text as human - is closer to one in 70, according to Spero's own tests. And that's before considering the arms race with AI labs making chatbots sound increasingly natural, plus 'humanizer' programs designed explicitly to disguise AI text.
I tested one such humanizer called Walter Writes AI. After having ChatGPT and Claude write brief articles, I ran them through Walter's rewording. ChatGPT's 'The numbers are no longer small enough to ignore' became 'The sheer size of these usage figures can no longer be ignored.' When I pasted the twice-baked output into Pangram, it invariably declared the text human-written. (Full disclosure: The Atlantic forbids using AI-generated text unless labeled as such, and I do not use AI for research.)
A New York City public high school teacher told me he's 'run some of my students' papers through Pangram, and it shows up as 100 percent human. And I don't think it is.' He knows what his kids are capable of and has ample reason to doubt Pangram. But accusing a student falsely carries high stakes: failure or resentment. 'The stakes are so high,' the teacher said, 'but our way of assessing what is AI-generated is still so unformed.'
Complicating matters further, Pangram's inner workings are opaque. The model was trained by feeding it mountains of human-written and bot-written examples - a book review from a magazine, then a ChatGPT-written review about the same book in the same magazine's style - until it learns to tell them apart. But Pangram can't point to specific evidence or patterns. 'The algorithm's inner workings are pretty uninterpretable,' Spero admitted. While he wants to make Pangram's 'AI-assisted' label more granular, he's 'still not sure how possible it is.' We risk layering dependence on yet another black-box algorithm.
Spero insists Pangram should 'never be the ending arbiter' but a starting point for investigation, and that the company investigates every reported error. He notes that smoke detectors and TSA scanners have base error rates too. The biggest problem, he argues, lies not in the technology but in what it's trying to detect: AI seeping haphazardly into written communication.
As AI-writing accusations escalate, reliance on Pangram - or whatever detector dethrones it - will only grow. Pangram can connect to Canvas, the popular education platform, scanning student submissions. With more than 10 million high schoolers and some 20 million undergraduates in the U.S., each submitting dozens of written assignments yearly, even a one-in-10,000 error rate would produce plenty of false accusations.
Nor is Pangram guaranteed to maintain its current ability. As chatbots and humanizers adjust, AI detection 'will wax and wane in its effectiveness for reasons we can't predict, at times we can't predict,' said Tim Requarth, a neuroscientist who teaches science writing at NYU. Any third-party assessments of Pangram's accuracy will be from weeks or months past - rendering them all but obsolete in the accelerating world of AI. Basing AI rules on detection reliability is like building a sandcastle at low tide.
All of this seems like a disaster in the making. The murkiness creates room to launch or deny accusations of nearly any sort. Earlier this month, tech journalist Taylor Lorenz was accused on X of using AI to write a Vanity Fair story. Spero investigated and found Pangram had erred. 'Thank god for edit history,' Lorenz told me. 'I'm so paranoid.'
'AIGenerated' and 'AIAssisted' can be easily confused, by accident or bad faith. Wall Street Journal editor James Taranto recently called Pangram a 'defamation machine,' claiming it falsely flagged three opeds. Two implicated authors admitted to using AI for revision, which Taranto considers 'inaccurate and unfair to characterize' as 'AI-generated.' Meanwhile, someone used Pangram to analyze Pope Leo's encyclical, suggesting senior Vatican officials might have used AI for portions - spawning headlines like 'Did the Pope Use AI to Write About the Dangers of AI?' (The Vatican didn't respond to comment, though a Vatican writer called the allegations '100 percent false' and said Leo drafted the encyclical with pen and paper.)
This recalls the plagiarism wars of 2023 and '24, when activists like Christopher Rufo mobilized accusations against academics and university leaders - most notably leading to Harvard president Claudine Gay's resignation. Many accusations were spurious, based on fairly useless plagiarism-detection algorithms. The AI-detection wars to come may be even more contentious.
Pangram, to be clear, is not useless. But that's exactly the problem: It's too easy to twist and contest its conclusions, especially when nobody agrees on which AI uses are ethical. Just like chatbots, AI-detection tools have become effective enough for widespread use but not reliable enough to fully trust. In this way, Pangram and other detectors are mirror images of the AI products they hunt.