In what can only be described as the academic equivalent of dropping a piano on someone's head and then asking them to review the sheet music, OpenAI this week released nearly 400 AI-generated mathematical results spread across more than 700 manuscripts. The reaction from the mathematics community ranged from "staggering" to "pure insanity" - and those were the polite ones.
The collection spans combinatorics, several branches of geometry, number theory, theoretical computer science, algebra, topology, probability and statistical mechanics, and mathematical physics. It's so vast that OpenAI felt compelled to publish guidance on how to navigate the sprawling GitHub repository, which is a bit like handing someone a map to a building you just set on fire.
"Just going over the entire list of abstracts is overwhelming," said Álvaro Lozano-Robledo, a professor of mathematics at the University of Connecticut. Several mathematicians told The Verge that simply working through the roughly 40-page table of contents and abstracts took them the better part of an hour. For those keeping score at home, that's one hour just to read the menu.
Sprinkled among the hundreds of manuscripts are formalizations in Lean, a programming language and proof assistant that allows results to be verified computationally. These formalizations have proven instrumental in assessing some of OpenAI's previous mathematical claims, giving researchers confidence that a claim is logically correct even if they don't fully understand the argument behind it - which is a bit like trusting a calculator without knowing what division is.
But the degree to which each result had been verified varied wildly. On GitHub, OpenAI acknowledged that the results are "at different stages of verification" and that "many, but not all, of the manuscripts have been formalized." As of writing, fewer than half the manuscripts in the collection appear to have been described formally. OpenAI said only 300 top-line results out of 719 manuscripts had been formalized - around 42 percent - and that it "will update the repository with more formalizations as we obtain them." So, roughly 58 percent of the mathematical apocalypse remains unverified. Sleep tight.
Several mathematicians complained to The Verge about the lack of formalization, particularly given the sheer number of results. Even when Lean code accompanies a result, evaluation isn't instantaneous. Researchers must check that the formalization actually proves what the result claims - another time-consuming process. Several digging through the papers said that even where computer-verifiable proofs had been provided, the quality was inconsistent and the statements they verified did not always appear to map neatly onto the claims in accompanying manuscripts.
Kevin Buzzard, a mathematics professor at Imperial College London, said he had identified numerous theorems in his area of work - algebraic number theory - of which only around six immediately "stood out." Few, if any, of those appeared to be formally verified in Lean. "Hence, I either have to read possibly-not-correct slop, or wait for others to do the same, or wait for someone to formalise them before I can say for sure that the results are even correct." Buzzard's concerns were echoed by numerous other researchers, who presumably had other things to do this week.
Buzzard was far from alone in worrying about AI "slop" - a shorthand for low-quality, frequently erroneous AI-generated material that increasingly crops up online and in the real world, including academic papers. As in other fields, mathematicians told The Verge they have seen a huge uptick in such material produced with tools like ChatGPT and Claude in recent years. Much of it is confusing, hard to read, and demonstrates little understanding of the subject; it is especially shoddy when it comes to crediting other researchers. OpenAI's previous mathematical write-ups were widely criticized by experts for their sloppy nature, particularly their poor or nonexistent attribution. In conversations with The Verge ahead of the release, several researchers had taken to calling the impending flood of papers the "slopocalypse," or similar variations on the theme.
Whether the feared "slopocalypse" actually materialized is difficult to say, largely due to the bewildering volume of material released. Early indications suggest OpenAI took more care with papers this time around, or at least with some of them. Several mathematicians told The Verge that their first impressions were far better than they had expected, though by their own admission that was hardly a high bar given the company's previous shoddy publications.
"It's a big mess. It can cause a huge collapse in the academic culture and simply kill most of the faculties. It's a social problem and it seems that the AI labs are completely ignoring this issue," one researcher said, presumably while staring into the middle distance.
But better does not necessarily mean good, let alone up to the standards usually expected of academic work making claims of this magnitude. With formal verification absent for many of OpenAI's claims, the quality of the accompanying papers becomes particularly important; they are the primary means by which mathematicians can confirm, understand, scrutinize, and contextualize the results.
Producing rigorous mathematical papers is difficult work under even the best of circumstances. Doing so at this kind of scale is a formidable undertaking. OpenAI's models are pumping out mathematics at a dizzying speed and across a broad range of specialities, far outstripping the capacity of its human staff. The company simply does not have the breadth of expertise or resources to properly scrutinize its findings at the cutting edge of mathematics. Researchers told The Verge that it shows.
Many described papers that were difficult, sometimes practically impossible, to follow. "The write-up of the problem I know best made little sense after a quick read," Brendan Hassett, a mathematics professor at Brown, told The Verge. "If this had been written by a person, I wouldn't spend any more time trying to understand it. Of course, this leaves 721 other preprints!" (Hassett said this before OpenAI retracted three papers, because of course it did.)
Some researchers told The Verge several papers they or their colleagues had noticed appeared to cover ground already trod by other mathematicians, though were wary of saying so publicly before they had a chance to properly review the material. Others pointed to the unusual brevity of the work, with results they would ordinarily expect to be developed over hundreds of pages compressed into a few dozen or less.
Nalini Joshi, a mathematics professor at the University of Sydney in Australia, said a quick search through the release revealed little that overlapped with her own work, but noted that some of the papers she examined "have short bibliographies." Given previous criticism of OpenAI's crediting practices, she said she is "wary that attribution in the papers may be lacking the complete story."
But for all the slop and uncertainty, the overarching consensus is that the release contains some genuinely impressive work. While stressing the difficulties assessing the volume of material - and the need to properly verify the results - numerous researchers The Verge spoke to said the work appeared to be of a very high caliber, despite shortcomings in its presentation. In a pre-AI world, they said, many of OpenAI's results would clearly have warranted publication in top-tier journals and could have been enough to secure an academic career for their authors. A handful were described as being the kind of work that could make a mathematician a serious contender for a Fields Medal, one of the discipline's highest honors.
"I either have to read possibly-not-correct slop, or wait for others to do the same, or wait for someone to formalise them before I can say for sure that the results are even correct," Buzzard reiterated, in case the point hadn't landed.
Stanford mathematician Jared Duker Lichtman said there were "tens" of results he would put in this category, including progress toward the Riemann hypothesis, a special case of the Hodge conjecture, and a solution to the four-dimensional Kakeya conjecture. These are major problems in mathematics. Riemann - arguably the most notorious unsolved problem in the entire discipline - and Hodge are both among the seven famed Millennium Prize problems. Respectively, they are concerned with the distribution of prime numbers and, very roughly, how complex geometric shapes can be understood in terms of simpler building blocks. Kakeya, meanwhile, roughly asks how little space is needed to rotate a needle or pencil in every direction. A proof to the three-dimensional version of Kakeya was among the achievements NYU mathematician Hong Wang was awarded a Fields Medal for earlier this year.
"It's not the case that these are just silly problems that no one's ever heard of," mathematician Scott Armstrong said. "Many of them are like very well-known problems that many people have tried for decades."
As the dust began to settle, mathematicians were left confronting a landscape that had abruptly changed around them. Many of them had just watched years of work and carefully laid research plans evaporate in an instant, or knew colleagues who had. Across the field, whether reactions were laced with excitement, dread, despair, or something in between, there was a profound sense of disorientation.
By the following morning, that mood had not altered much. Speaking by phone as he walked through Paris, Armstrong imagined that, had the Sorbonne not been closed amid ongoing student protests and his colleagues were gathered around their usual coffee machine, the atmosphere would have been rather "solemn."
In Scotland, Colva Roney-Dougal, a professor of mathematics at St Andrews University, described a similarly bleak atmosphere among colleagues and students. The deluge felt somewhat "horrific," she said, but its arrival brought some relief after weeks of uncertainty. "I have a bunch of friends and colleagues whose grant proposals have just been wiped out," she said.
Armstrong said he knew of one group whose entire research program was practically "wiped out" by the release. Tristan Buckmaster, an NYU mathematician who was at the center of OpenAI's earlier dispute over the Navier-Stokes problem, meanwhile, said he had already heard of "three people who had entire research programs obliterated."
Similar stories surfaced repeatedly in The Verge's conversations with mathematicians, though the disruption was concentrated heavily in some areas of the field. Francesco Fournier-Facio, a professor of mathematics at Heriot-Watt University in Scotland, said researchers in probability, combinatorics, and theoretical computer science appeared "particularly in shock," adding that some regions of his field, group theory, had been "bulldozed."
Armstrong and other researchers said some areas appeared to have been aimed at with almost military precision. "There were definitely some targets," said Armstrong. He singled out work in the area Wang received the Fields Medal for this year, as well as several Millennium Prize problems. A cluster of results in mathematical physics, he said, makes it clear OpenAI - and here the sentence trails off, presumably because the rest of us are still trying to process the first 400 results.