Benchmaxxing Just Got Busted: AI Benchmark Overhaul Drops Kimi, Widens the Open-Source Gap

What Happened: The Leaderboard Got Rewritten Overnight

On October 28, 2026, benchmark provider Artificial Analysis released Intelligence Index v5 — its largest methodological overhaul since January — and the AI model leaderboard immediately rearranged itself. The headline casualty: Beijing-based Moonshot AI's flagship Kimi K3.1 Pro collapsed from 4th place to 9th under the new scoring, the model that just weeks ago topped Terminal Bench 4.0 and was widely hailed as the strongest open-weight challenger ever shipped.

The fix was simple and brutal: roughly 60% of the index's test questions are now private, with held-out answers, specifically designed to neutralize "benchmaxxing" — the practice of training models against publicly available test prompts hosted on GitHub and Hugging Face. Under v5, models can no longer study for the exam they're about to take.

The reshuffle also crowned a new #1: Google's unreleased Gemini 4 Carbon at 59 points, the cybersecurity-first frontier model announced just a day earlier, ahead of Claude Opus 5.5 (58) and GPT-6 Astra (57). But the bigger story for anyone choosing AI tools isn't who won — it's how much of the old ranking was contaminated in the first place.

What Is Benchmaxxing — and Why It Fooled Everyone

Benchmaxxing is the AI industry's version of teaching to the test. Public benchmarks like MMLU, GPQA, and coding evals publish their questions or see them leak onto sites where training data is scraped. A lab that wants a pretty leaderboard number can quietly include benchmark items in pretraining or fine-tuning mixes. The model then "scores" far above its real capability — and buyers make decisions based on a number that measures memorization, not intelligence.

The v5 coding component shows how seriously Artificial Analysis took the problem. Coding now carries 25% of the total weight, with 15% coming from Cognition's FrontierCode 2.0 — a benchmark where grading asks whether real open-source repository maintainers would merge the pull request the model produced. There are 132 tasks, and a single failing blocking check scores the whole task as zero. No partial credit for plausible-looking code that doesn't run.

That standard is closer to how you actually experience a model inside a tool like Claude Code or ChatGPT: either the change works, or you're debugging it yourself at midnight.

Winners and Losers in the v5 Reshuffle

The quiet winner is DeepSeek. Its V4.1 Flash (the October 17 release) posted 28% on FrontierCode 2.0 — the highest score of any model with actually published weights, ahead of Kimi K3.1 Pro's 25.5% and even Gemini 4 Argon's 25.7%. If you're betting on open-source — running DeepSeek locally or through a cheap API — the reshuffle says your money was on the right horse after all.

The Open-Source Gap Just Tripled — Or Did It?

The most consequential number in the whole update: under v4, proprietary frontier models led open-weight models by 3 points. Under v5, that gap exploded to 12 points.

Read that carefully. The models didn't get worse overnight — the measurement changed. What the widening really means is that a chunk of the open-weight ecosystem's apparent parity was benchmark contamination. When you can't study the test anymore, the proprietary frontier pulls clearly ahead on genuinely held-out questions.

Two caveats before you cancel your self-hosting plans. First, the gap is measured at the frontier — for everyday writing, summarization, and routine coding, open models remain dramatically cheaper and entirely capable. Second, Kimi's position may partially recover: Moonshot has promised to publish K3.1 Pro's full weights by November 4, and the index currently flags it as "weights not yet released." A fair verdict on the open-weight frontier may still be a week away.

How to Pick AI Tools When Benchmarks Lie

If leaderboard positions can swing five places on a methodology change, rankings alone are a shaky basis for picking your stack. Four rules that survive any index version:

1. Judge the tool, not just the model. The same model performs differently behind different harnesses — Carbon in Google's Antigravity trails Opus 5.5 in Claude Code on the Coding Agent Index despite winning the overall intelligence crown. Workflow, context handling, and tool integration often matter more than raw model score.

2. Prefer private-eval evidence. When a vendor quotes benchmark wins, check whether the benchmark is public (contamination-prone) or private. Post-v5 numbers are simply worth more than pre-v5 numbers, and comparisons across the two are invalid.

3. Run your own test. The best benchmark is your actual workload. Most serious tools — Claude, Grok, ChatGPT, DeepSeek — have free tiers or cheap APIs. Twenty real tasks from your own work will beat any index.

4. Match the model to the job. The v5 data shows specialization everywhere: Carbon dominates reasoning but not coding agents; DeepSeek leads open-weight coding; Claude Opus 5.5 keeps the coding-agent crown. Multi-model routing — using different models for different tasks — is now the cost-effective default, not a power-user trick.

Frequently Asked Questions