Let’s be honest for a second. We’ve all been there. You ask an AI a question, and it gives you an answer that sounds incredibly confident, articulate, and… completely wrong. It’s that weird mix of impressive and infuriating that defines where we are with AI right now.
For a while, we’ve been measuring these models with benchmarks that test how well they can code, follow instructions, or browse the web. And they’re getting really good at those things! But there’s been a huge, glaring hole in our testing: how often are they just making stuff up? How factual are they, really?
For anyone building products in finance, law, or medicine—where a "small hallucination" can have massive consequences—this isn't just an academic question. It's everything.
Well, the folks at Google just dropped a reality check on the entire industry. They released a new evaluation framework called the FACTS Benchmark, and it’s basically a truth serum for large language models. The results are a massive wake-up call, and the big headline is this: no model is cracking a 70% accuracy score.
That's right. The smartest AIs in the world are, on average, still getting things wrong about a third of the time. This isn't just a small gap; it's a "factuality wall" that every single developer and product manager needs to understand.
So, What's Actually on This Test?
Before we get into the juicy leaderboard drama, let’s quickly break down what FACTS is actually measuring. This isn't your standard multiple-choice quiz. Google designed it to mimic the real-world ways AI fails.
Think of it as four different exams:
- The "Closed-Book" Test (Parametric): Can the AI answer a trivia question correctly using only the knowledge it was trained on? Basically, how good is its internal memory?
- The "Open-Book" Test (Search): Can the AI use a web search tool to find, understand, and synthesize information to answer a question? This is a huge test for any RAG (Retrieval-Augmented Generation) system.
- The "Visuals" Test (Multimodal): Can the AI look at a chart, a diagram, or a photo and accurately tell you what’s in it? Can it read the data off a bar graph without hallucinating?
- The "Stick to the Script" Test (Grounding): If you give the AI a specific document (like your company's HR policy), can it answer questions using only that information, without adding its own creative flair?
Google released a bunch of these test questions to the public, but they’re keeping a private set locked away at Kaggle. This is a smart move to prevent people from just training their models on the test itself—a classic cheating problem in the AI world.
The Leaderboard: It's a Game of Inches, But Everyone's Losing
Alright, here’s the part you’ve been waiting for. Who’s at the top of the class? According to the initial results, Google’s own Gemini 3 Pro takes the lead.
Here’s a quick look at the overall scores:
- Gemini 3 Pro: 68.8%
- Gemini 2.5 Pro: 62.1%
- OpenAI’s GPT-5: 61.8%
- Grok 4: 53.6%
- Claude 4.5 Opus: 51.3%
Now, looking at that, you might think, "Okay, Gemini is the clear winner." But the overall score is a vanity metric. It hides the most important details. The real story for those of us building things is in the sub-scores.
Let me show you what I mean. Here are the scores for the two most critical areas for enterprise use: using search tools (RAG) and understanding images (Vision).
| Model | Overall Score | Search (RAG) | Multimodal (Vision) | | :--- | :--- | :--- | :--- | | Gemini 3 Pro | 68.8% | 83.8% | 46.1% | | Gemini 2.5 Pro| 62.1% | 63.9% | 46.9% | | GPT-5 | 61.8% | 77.7% | 44.1% | | Grok 4 | 53.6% | 75.3% | 25.7% | | Claude 4.5 Opus | 51.3% | 73.2% | 39.2% |
See what’s happening here? The story gets a lot more complicated, and frankly, a lot more useful.
For Builders: RAG Isn't Optional Anymore
If you’re building any kind of system that needs to access up-to-date, factual information—like an internal knowledge bot or a research assistant—the Search benchmark is your bible.
And look at the data! Gemini 3 Pro scores a fantastic 83.8% on Search tasks. That’s a solid B. But on the "Closed-Book" test (Parametric), where it relies on its own memory, it only scores 76.4%. This is the most important data point in the whole report, in my opinion.
It’s a massive validation of the RAG architecture that so many of us have been building. The message is crystal clear: Do not rely on your model’s internal memory for critical facts. Hooking your AI up to a search tool or a vector database isn’t a fancy add-on; it's the only reliable way to get factually-grounded answers.
The Real Shocker? AI Can't Reliably Read Charts.
Okay, now for the part that should make every product manager a little nervous. Look at those Multimodal scores. They are, to put it bluntly, terrible.
The best model, Gemini 2.5 Pro, is only getting it right 46.9% of the time. That’s worse than a coin flip. The benchmark included tasks like reading data from a simple chart or identifying objects in a diagram.
This is a huge red flag. If your product roadmap includes a feature where an AI automatically pulls data from invoices, interprets financial charts, or analyzes medical diagrams without a human in the loop, you need to pause. Right now, you’d be introducing a massive error rate into your system. Multimodal AI is incredibly powerful, but these numbers show it’s just not ready for unsupervised, high-stakes work.
So, How Do You Use This Without Getting Fired?
The FACTS benchmark is going to become the new standard. When you're deciding which model to use in your next project, you can't just look at the marketing hype. You need to look at these numbers and match the model's strengths to your specific job.
Here’s a simple cheat sheet:
- Building a customer support bot? You care most about the Grounding score. You need a bot that sticks to your official documents. Interestingly, Gemini 2.5 Pro (74.2%) actually beats the more powerful Gemini 3 Pro (69.0%) here.
- Building a research assistant? You should be obsessed with the Search score. Gemini 3 Pro is the clear leader for now.
- Building an image analysis tool? Look at those Multimodal scores and proceed with extreme caution. Plan for a human-in-the-loop workflow for the foreseeable future.
The Google team said it best: "All evaluated models achieved an overall accuracy below 70%, leaving considerable headroom for future progress." That’s a very polite way of saying, "Folks, we’ve got a long way to go."
For now, the message is simple. The models are getting smarter every day, but they are not oracles. They are powerful, flawed tools. We have to design our systems with the assumption that the raw output might just be wrong. The era of "trust but verify" isn't ending anytime soon. In fact, it's just getting started.
