Way back in the summer of 2017, Google researchers dropped a paper called “Attention Is All You Need,” introducing the transformer - a neural network that turned out to be spectacularly good at chewing through long sequences of text. Nine years later, transformers are the engine inside every major large language model (LLM). “The entire AI industry is built on transformers,” says Justin Dangel, CEO of AI startup Subquadratic, calling them one of the most important innovations in computer science history. But even the greatest hits start to feel tired after a while.

Transformers are showing their age. Recent LLM advances - like reasoning models that scribble notes to themselves and handle massive inputs - aren't neat extensions of the core tech; they're duct-tape patches over fundamental flaws. So a growing pack of scientists and engineers are asking: what's next? LLMs aren't going anywhere, but how they're built is up for grabs. MIT Technology Review has dubbed this future generation “LLMs+,” and a wave of startups is charging in, hoping to push the boundaries of this boomtown tech. Some will fail, but they've got everything to play for and far less to lose than today's front-runners.

First, the problem. Transformers rely on dense attention, which compares every word (or token) with every other word via multiplication. It captures meaning brilliantly, but the compute scales horribly: a 10,000-word document might need 50 million multiplications. That's why LLMs suck up so much power. OpenAI's president Greg Brockman says the company will spend $50 billion on computing this year, and the International Energy Agency predicts data center electricity use will double by 2030. Transformers also struggle with large context windows - keeping track of a library, a codebase, or agent outputs - and reasoning models add more data to juggle. As LLMs grow, transformers become a bottleneck.

Here are four wild ideas to fix it.

**Idea 1: Sparse Attention, Subquadratic Style**

Miami-based Subquadratic claims it's cracked sparse attention - computing only some word pairings instead of all - without losing meaning. Its model, SubQ, allegedly rivals top LLMs on search and coding tasks. Skeptics remain, but thousands have joined the waitlist. The company says it figures out on the fly which words matter, and plans a wide release soon.

**Idea 2: Power Retention from Manifest AI**

San Francisco's Manifest AI ditches attention for “power retention,” which stores only the most relevant info and provides a rolling summary of the context window, dropping less relevant bits as new data arrives. The concept is a decade old, but Manifest claims it's finally competitive with transformers. It turned the open-source coding LLM StarCoder into PowerCoder and released Brumby, which it says rivals Alibaba's Qwen. Cofounder Carles Gelada sees applications from analyzing hours-long videos to agents that stay on task for weeks.

**Idea 3: Liquid Neural Networks from Liquid AI**

MIT spinout Liquid AI pairs transformers with liquid neural networks - inspired by worm brains - to create “liquid foundation models” (LFMs). These are far smaller and more energy-efficient, running on a $50 Raspberry Pi. They're free for organizations with under $10 million annual revenue, and have racked up nearly 34 million downloads, says CEO Ramin Hasani. The models match rivals four times bigger, including versions of Qwen and Google's Gemma. The secret sauce: an AI that designs the models, settling on a hybrid of 20% transformers and 80% liquid networks. Hasani marvels that the brain runs on 20 watts: “We can get a lot more innovative.”

**Idea 4: Diffusion from Inception**

Inception, based in Palo Alto, generates whole sentences at once using diffusion - the tech behind image and video models. Diffusion LLMs, like Mercury 2, claim to match GPT-4 performance but 10 times faster. Cofounder Stefano Ermon, a Stanford researcher, figured out the math to make diffusion work with text in 2024, after training a model that beat GPT-2's performance at 10x speed. Google is experimenting too with Diffusion Gemma, but Ermon isn't worried: “This is the future.”

**The Extreme: Pathway's Dragon Hatchling**

Pathway, also Palo Alto-based, wants to free LLMs from language entirely. Its model, Dragon Hatchling (named after Terry Pratchett's dragons that materialize if you think about them hard enough), solved over 97% of 250,000 hard sudoku puzzles - while leading LLMs failed any. The point: there are still crucial problem classes where LLMs fail, and if we want genuine reasoning, we need to think beyond transformers.

So there you have it: sparse attention, power retention, worm brains, and diffusion - all vying to be the next big thing. The only currency that matters, says Ermon, is “intelligence per dollar.” And if these startups are right, the transformer's reign might finally be challenged - just in time for the data centers to double their electricity use.