ARC Prize Rejects OpenAI’s 99.9% AGI Score — GPT-6 Astra Reran at 62.7% on a Neutral Harness

The Headline Score That Didn’t Survive Contact With the Referee

When OpenAI launched GPT-6 Astra on September 3, 2026, the number everyone quoted was 99.9% on ARC-AGI-3 — the abstract-reasoning benchmark built by the ARC Prize Foundation specifically to resist memorization and brute-force gaming. Three days later, Nvidia CEO Jensen Huang cited that exact figure on X to declare that “AGI has arrived,” in a post that cleared 10 million views by the following Monday.

Now the referee has weighed in. ARC Prize reran GPT-6 Astra on ARC-AGI-3 through the same neutral harness it uses for every other lab’s model — and the score came back at 62.7%. That is still the highest verified result on the benchmark, a genuine leap from the near-zero scores of early 2026. But it is more than 37 points below the number used to announce the arrival of artificial general intelligence. ARC Prize’s verdict on the 99.9% was blunt: not valid, because it wasn’t produced under the conditions every other model has to clear.

Where the 37 Points Went

According to reporting on the benchmark revisions, OpenAI’s 99.9% came from its own custom “tools-enabled” harness — a configuration that lets the model preserve reasoning state across actions in a way the standard test setup does not, at a cost of nearly $19,000 per run. No other lab’s model is evaluated under that configuration, so the 99.9% is not comparable to any other score on the board. On ARC Prize’s leaderboard in late October, GPT-6 Astra’s lower-compute variants sit at 59.3% and 54.8%.

Two details make the story sharper. First, OpenAI quietly revised five published benchmark figures after the launch post went up. Second, ARC Prize declined to call Astra’s progress AGI at all — a pointed refusal from the organization whose test was invoked to say it had.

The uncomfortable part for the rest of us: none of this required deception. A tools-enabled configuration is a legitimate engineering choice. The problem is that a number generated under special conditions was presented next to — and compared against — scores produced under standard ones. That is exactly the ambiguity that makes benchmark-driven tool buying so treacherous.

The Harness Lesson: Same Model, Wildly Different Scores

The ARC dispute is the loudest example of a pattern that kept repeating this month: the scaffolding around a model can swing results as much as the model itself.

• On ARC-AGI-3, the open-source Prime Agent harness pushed public scores from roughly 30% — where Claude Opus 5 sat alone — past 95% on the same class of model, without changing the weights.
• Researchers at Shandong University ran Claude Code against a bare-bones ~100-line Python harness on 492 SWE-bench Verified tasks. Accuracy came within two points; per-task cost varied by up to 3x on the same model, driven by context bloat (16,581 initial tokens vs. 829). On the 45 hardest tasks, a median 13% of results flipped pass-to-fail purely from switching harness.
• When models are close in raw capability, the harness — not the model — decides both your score and your bill.

The practical translation for tool shoppers: when a vendor publishes a benchmark, ask what configuration produced it. A coding agent’s leaderboard number reflects its default scaffolding, context management, and tool loop — all things you may override, pay for, or disable in your own workflow.

Exhibit B: Replit Picked Its Model on Price, Not Benchmarks

The same week, The Information reported that Replit moved more than $100 million a year of AI spend from Anthropic to OpenAI — after OpenAI offered a discount of more than 30% and Anthropic discontinued its own. Replit’s CEO confirmed OpenAI models are now the default in its agent products.

A nine-figure account changing hands over a withdrawn discount tells you something benchmarks can’t: at the frontier, the top models are now close enough on capability that procurement decides on price, latency, and terms. The gap between GPT-6 Astra, Claude Opus 5.5, and Gemini 4 Argon on real work is smaller than the gap between their marketing claims. Your own tasks — not their harnesses — are the tiebreaker that matters.

How to Actually Evaluate an AI Tool in 2026

Whether you’re choosing between ChatGPT, Claude Code, Cursor, Gemini CLI, or OpenCode, the same discipline applies:

• Ask what harness produced the score. Vendor-run, tools-enabled, or high-compute configurations are not comparable to neutral-board numbers. If the setup isn’t disclosed, treat the score as marketing.
• Run your own eval before you commit. Take 10–20 real tasks from your workflow, run them across two or three tools, and score the outputs yourself. An afternoon of testing beats any leaderboard.
• Measure cost per completed task, not per token. The SWE-bench harness study found 3x cost differences on identical models. Context bloat, not model pricing, is often the real bill.
• Re-test when the harness changes. A coding agent update, a new default context strategy, or a model swap can flip results on hard tasks in either direction.
• Watch for revisions. OpenAI revised five published figures post-launch. Screenshot the claim when you first see it — and check whether it still stands before you renew.

Frequently Asked Questions