Imagine spending years creating something that fundamentally changes the world. You co-author a paper so influential it gets cited over 100,000 times. You even coin the name for your invention: the "transformer." This technology becomes the engine behind ChatGPT, Claude, and basically every major AI model you've ever heard of. You're at the top of your field.
Now, imagine standing on a stage and telling everyone you're "absolutely sick" of your own creation.
That’s exactly what Llion Jones, one of the architects of the modern AI boom, did at the TED AI conference. In a refreshingly candid talk, the man who helped give us generative AI dropped a bombshell: he thinks the entire field has become dangerously stuck, and he's personally moving on from the very technology that made him a legend. It’s a bit like if J.R.R. Tolkien announced he was tired of fantasy and was only going to write minimalist poetry from now on. When a creator disowns their creation, you lean in and listen.
The Paradox: How Billions of Dollars Are Killing AI's Creativity
You’d think that with the firehose of cash, talent, and hype flooding into artificial intelligence, we’d be in a golden age of creative research. Jones argues the exact opposite is happening. The immense pressure from venture capitalists demanding returns and researchers desperate to publish before they get "scooped" has created a toxic environment of conformity.
He describes a community suffering from a painful paradox: more resources have somehow led to less genuine creativity. Instead of exploring wild, unconventional ideas, researchers are playing it safe.
"If you're doing standard AI research right now," Jones explained, "you kind of have to assume that there's maybe three or four other groups doing something very similar, or maybe exactly the same." This frantic race to the finish line, he says, "damages the science, because people are rushing their papers, and it's reducing the amount of creativity."
Stuck on "Exploit," Forgetting to "Explore"
Jones borrows a concept from AI itself to explain the problem: the "exploration versus exploitation" trade-off.
- Exploitation is when you stick with what you know works and try to optimize it for small gains.
- Exploration is when you take a risk and search for a completely new, potentially much better solution.
A healthy system needs a balance of both. But right now, Jones argues, the AI industry is all exploitation. Everyone is tweaking the same transformer architecture, hoping for a slightly better benchmark score. "We are almost certainly in that situation right now in the AI industry," he warned. We're so busy mining the same gold vein that we're completely missing the diamond mine just over the hill.
He worries we're repeating history. Before transformers, everyone was obsessed with fine-tuning a different architecture called recurrent neural networks (RNNs). Then, "Attention Is All You Need" came along and made all that incremental work seem quaint and irrelevant overnight. Are we doing the same thing today, polishing a technology while its successor is waiting in the wings, undiscovered?
The Secret Origin Story of the Transformer
So, if today's high-pressure environment is the problem, what's the solution? Jones points to the very conditions that allowed the transformer to be born in the first place—and it’s a world away from the current landscape.
The "Attention Is All You Need" paper wasn't the result of a top-down corporate mandate or a frantic race against a competitor. It was, in his words, "very organic, bottom up." It started with casual chats over lunch and "scrawling randomly on the whiteboard."
Here’s the critical part: the team at Google had freedom.
"We didn't actually have a good idea, we had the freedom to actually spend time and go and work on it," Jones recounted. "And even more importantly, we didn't have any pressure that was coming down from management. No pressure to work on any particular project, publish a number of papers to push a certain metric up."
That's the magic ingredient that's gone missing. Today, even researchers hired for million-dollar salaries feel the squeeze. Jones posed a sharp question: when those star researchers start a new job, do they "feel empowered to try their wild ideas... or do they feel immense pressure to prove their worth and once again, go for the low hanging fruit?" We all know the likely answer.
Sakana AI: A Bet on Freedom Over Formula
Jones isn't just complaining from the sidelines. He's putting his money where his mouth is. As the co-founder and CTO of Tokyo-based Sakana AI, he’s explicitly trying to recreate that "pre-transformer" environment of pure exploration.
"I personally made a decision in the beginning of this year that I'm going to drastically reduce the amount of time that I spend on transformers," he declared. "I'm explicitly now exploring and looking for the next big thing."
At Sakana, the guiding principle is a mantra from engineer Brian Cheung: "You should only do the research that wouldn't happen if you weren't doing it."
It's a powerful filter. It pushes the team away from crowded, mainstream topics and toward the truly novel. One of their projects, a "continuous thought machine" inspired by brain synchronization, is a perfect example. The employee who pitched it told Jones that at any other top lab, he would've been told not to waste his time. At Sakana, he was given a week to play with the idea. That project ended up being spotlighted at NeurIPS, one of the world's top AI conferences.
Jones even thinks this culture is a secret weapon for recruitment. In a world of astronomical salaries, the promise of genuine intellectual freedom can be the ultimate perk. "Talented, intelligent people, ambitious people, will naturally seek out this kind of environment," he says.
Is the Transformer a Victim of Its Own Success?
Perhaps Jones's most provocative idea is that the transformer architecture may be too good for its own good. It's so powerful and flexible that it has created a gravitational pull, preventing researchers from achieving escape velocity and exploring new orbits.
"The fact that the current technology is so powerful and flexible... stopped us from looking for better," he mused. "It makes sense that if the current technology was worse, more people would be looking for better."
It's a stunning thought. Could AI's biggest breakthrough be the very thing holding back the next breakthrough?
He’s careful to clarify that work on transformers isn't pointless. There's still immense value to be squeezed out of the current paradigm. "I'm just saying that given the amount of talent and resources that we have currently, we can afford to do a lot more" exploration. The balance is just all wrong.
His message isn’t about tearing down his own legacy. It’s a call for the entire community to collectively shift its mindset. This isn't a zero-sum game between labs. "Genuinely, from my perspective, this is not a competition," Jones concluded. "We all have the same goal. We all want to see this technology progress so that we can all benefit from it."
His final plea is for everyone to turn up their "explore dial" and, crucially, to "openly share what we find." If we do that, he believes, we can all reach our shared goal much, much faster. As the industry grapples with signs that simply making models bigger is yielding diminishing returns, Jones’s warning from the inside couldn't be more timely. The next world-changing idea might be out there, just waiting for someone with the freedom to find it.
