Have you ever wondered what’s really going on inside the black box of models like Anthropic's Claude? They're incredibly powerful, but the companies behind them are often tight-lipped about the exact architecture. They release the model, but not the blueprints.
Well, the AI research community loves a good mystery. And for a while now, they've been theorizing about a specific model: Claude Mythos.
Enter Kye Gomez, who just dropped an open-source project on GitHub called OpenMythos. Now, let's be crystal clear about what this is. It's not a leak. It's not a fine-tuned copy. It's something way more interesting: a first-principles, theoretical reconstruction of what Claude Mythos might be. It’s a hypothesis, written entirely in PyTorch code. And because it's code, it's a theory we can actually run, test, and even break. That’s what makes this so exciting.
So, What's the Big Idea? A "Looped" Transformer
The core claim of OpenMythos is that Claude Mythos isn't built like the models we're used to. It's not another GPT or LLaMA-style transformer that just stacks layer after layer after layer.
Instead, the theory is that it’s a Recurrent-Depth Transformer (RDT), sometimes called a "Looped Transformer."
Think of it this way. A standard transformer, like GPT-4, is like reading a massive book. You process the information page by page, chapter by chapter, one after the other. Each chapter (layer) has its own unique words and sentences (weights). To make the book "smarter," you just add more chapters, making it thicker and heavier (more parameters).
An RDT is completely different. It’s more like refining a single, complex paragraph. You write a draft, then you read it over, make some changes, and read it again. You keep looping over the same paragraph, improving it with each pass. The "depth" of your understanding doesn't come from how many pages are in the book, but from how many times you’re willing to re-read and refine that core idea.
That’s an RDT. It takes a single computational block with a fixed set of weights and applies it over and over again in a loop. Reasoning depth isn't about how many parameters you can store; it's about how many iterative loops you run when you ask it a question.
How OpenMythos Pieces It All Together
The OpenMythos architecture breaks this down into a simple three-act structure:
- The Prelude: A few standard transformer layers that process your initial input once. They get the ball rolling.
- The Recurrent Block: This is the heart of the machine. It's the computational core that gets looped up to 16 times. This is where the "thinking" happens.
- The Coda: A final set of standard transformer layers that run once at the end to generate the final answer.
The magic is all in that Recurrent Block. At every single loop, two crucial things happen. First, the model's current "thought" (its hidden state) is updated. Second, the original input from the Prelude is re-injected.
This re-injection is a clever little trick. Without it, after a few loops, the model might forget what the original question was. It's like constantly reminding someone in a deep conversation, "Hey, just remember, we started out by talking about this." It keeps the whole process anchored to the initial prompt.
The Secret Sauce Inside the Loop
But there's more going on inside that recurrent block. It’s not just a simple loop.
First, it doesn't use a standard feedforward network (FFN). Instead, OpenMythos proposes it uses a Mixture-of-Experts (MoE) layer, similar to the one in the DeepSeekMoE model. Imagine you have a huge team of specialists (the "experts"). For any given task, you don't need the whole team. You just need to call on the two or three specialists who are perfect for that specific job. That's MoE. It makes the model incredibly efficient because it only activates a tiny fraction of its total experts for any given token.
And here’s the kicker: the router that picks the experts selects a different subset of experts at each loop depth. So even though the model is re-using the same core weights, each pass through the loop is computationally unique. You get the depth of looping with the breadth of having many specialized experts.
"Thinking" Without Actually Typing
This is probably the most mind-bending part of the whole idea. In this architecture, all the reasoning happens in a continuous latent space.
What does that even mean?
When you use something like Chain-of-Thought prompting, you're essentially asking the model to "think out loud." It generates words and sentences as intermediate steps, then reads its own writing to continue the thought process. The reasoning is externalized into tokens.
An RDT doesn't do that. There's no text generated between the loops. The model "thinks" entirely in terms of abstract mathematical vectors. It's the difference between solving a math problem on a whiteboard for everyone to see versus solving it entirely in your head.
This has a massive advantage. Researchers have shown that each loop in an RDT is functionally equivalent to one step of Chain-of-Thought, but it's happening internally with rich, continuous vectors instead of clunky, discrete words. This allows it to explore multiple possible reasoning paths at once.
Even better, it helps the model generalize. A standard transformer trained on, say, 5-step reasoning problems will completely fail if you give it a 10-step problem. It simply wasn't built with enough layers. An RDT, on the other hand, can just run more loops. It can dedicate more compute to harder problems at inference time without needing to be retrained.
Keeping the Model Stable (And Sane)
Now, if you've ever worked with recurrent systems, you know they can be notoriously unstable. Left unchecked, a looped model can run into two major problems.
- Residual Explosion: The internal state can grow exponentially with each loop until the numbers are so huge they just become noise. The model basically has a mathematical meltdown.
- Overthinking: The opposite problem. After a certain number of loops, the model can actually go past the correct answer and start degrading its own prediction. It thinks so hard that it confuses itself.
OpenMythos proposes solutions for both. To prevent the "explosion," it uses a stability constraint borrowed from another architecture called Parcae. It basically puts a mathematical governor on the engine, guaranteeing it can't spiral out of control.
To solve "overthinking," it uses a technique called Adaptive Computation Time (ACT). This is brilliant. It’s a small, learned mechanism that lets each part of the input decide for itself when it has had enough computation. Easy-to-process tokens can "halt" early, while more complex parts of the problem get more loops. It dynamically allocates compute where it's needed most.
Why This All Matters: Punching Way Above Its Weight
Okay, so it’s a cool academic idea. But what’s the payoff?
Parameter efficiency. Massive parameter efficiency.
Research from the Parcae paper (where some of these ideas come from) showed that a 770-million-parameter RDT could match the performance of a 1.3-billion-parameter standard transformer. That's nearly the same capability with almost half the parameters.
This could fundamentally shift how we think about scaling AI. For years, the dominant idea has been "more parameters = more power." OpenMythos suggests a different path. It says that reasoning depth can scale with inference-time compute (how many loops you run), not just the number of parameters you stored at training time. The future might not be about building bigger and bigger models, but about building models that can think harder when they need to.
So, What Do We Get from OpenMythos?
Whether or not this is actually how Claude Mythos works is almost beside the point. Kye Gomez and the OpenMythos project have given the AI community something incredibly valuable:
- A full, working PyTorch implementation of this RDT hypothesis.
- Built-in tools to handle the tricky stability problems of training looped models.
- A reproducible baseline for researchers to study these kinds of architectures.
It’s a complete, runnable toolkit for exploring a path less traveled in AI. It gives us a concrete alternative to just throwing more GPUs at bigger and bigger traditional transformers. And in a field that can sometimes feel like it's stuck in a rut of "scale is all you need," a fresh, compelling, and fully-coded idea like this is a breath of fresh air. It’s a reminder that there are still fundamental architectural mysteries to solve, and the next big leap might come from a completely different direction.
