A 10-Year-Old Neural Network Beat GPT-6 Astra and Claude Fable 5.1 at Skin Cancer Diagnosis

What Happened: 200 Lesions, Three Contestants

A dermatologist spent a weekend running one of the more bracing AI experiments of the year: he took 200 biopsy-proven skin lesions from his own practice and showed every one of them to three very different systems — OpenAI's GPT-6 Astra, Anthropic's Claude Fable 5.1, and a fine-tuned Inception v3 convolutional network, an image-classification architecture that dates back to 2015.

The question was simple: for each lesion, what's the diagnosis? No hints, no multiple choice — just the image, and whatever each system could make of it.

The result, published as a blog post on October 26, is a useful corrective to a year of "frontier model does everything" headlines. The decade-old specialist network, fine-tuned on exactly this kind of data, out-diagnosed both frontier chatbots by a wide margin. But the story doesn't end there — because on the metric that actually matters in medicine, the loser won.

The Scoreboard: Old Network 125, Frontier Models 85

The fine-tuned Inception v3 network named the correct diagnosis for 125 of the 200 lesions — a 62% hit rate. GPT-6 Astra and Claude Fable 5.1 each landed around 85 correct diagnoses, or roughly 42%, less than half of lesions despite being among the most capable AI systems ever shipped.

The failure pattern matters as much as the score. GPT-6 Astra kept defaulting to basal cell carcinoma — a common, slow-growing, rarely dangerous skin cancer — including for 14 actual melanomas, the lesion type where a missed diagnosis can kill. When a generalist doesn't know what it's looking at, it drifts toward the statistically safest-sounding answer. In dermatology, that drift is exactly backwards.

The Twist: The Winner Was Also the Most Dangerous

Here's where the experiment flips. The dermatologist then looked at triage: for the 30 melanomas in the set, would each system have sent the patient home?

• At its default operating point, the old network would have discharged 18 of the 30 melanomas — a 60% miss rate on the most lethal diagnosis in the set.
• The chatbots, cautious by design, would have sent home only 3 and 4 of the 30.

So the "smarter" system is the one that would have gotten patients killed, while the "dumber" chatbots over-referred almost everything — annoying, expensive, but survivable. Recognizing a lesion and safely triaging it are two different scores, and the experiment accidentally measured both.

That distinction maps perfectly onto how you should evaluate any AI tool, medical or not: raw accuracy on the headline task versus behavior when it's uncertain. The second one is the product. The first is a benchmark.

Why Generalists Lose to Specialists

None of this means frontier models are overhyped. It means they're general-purpose reasoners, and the dermatologist's CNN is a single-purpose instrument trained end-to-end on exactly this distribution of images. A few structural reasons the specialist wins here:

• Training distribution. The fine-tuned network has seen tens of thousands of labeled lesions. The chatbots have seen the whole internet once, thinly — dermatoscopic images are a rounding error of their pretraining data.
• Objective function. A classifier is optimized to place every input into one of N diagnostic buckets. A chatbot is optimized to produce a plausible, helpful-sounding continuation — which is not the same skill.
• Calibration. The chatbots' bias toward common, benign-sounding diagnoses is a side effect of language-model priors. The network has no language priors to be biased by.
• Cost and latency. The 2015-era network runs on a laptop GPU for a fraction of a cent per image. Frontier inference is orders of magnitude more expensive — for a worse answer on this task.

The pattern repeats across industries: specialized research tools still beat generalist chatbots at literature review; dedicated medical illustration AI beats prompting a frontier model for clinical imagery. "Newest and biggest" is a bad selection criterion. "Trained for this exact job" usually isn't.

What This Means When You Pick AI Tools

If you're choosing AI tools for a business — or just for yourself — this experiment is a compact checklist:

The Caveats — Read Them

Before anyone retires their dermatologist: this is one doctor's blog post, not peer-reviewed research. Each chatbot was run once per lesion, with no prompt tuning, no vision-specific system prompts, and no retries — frontier models are sensitive to all three. The lesion set came from a single practice, so the distribution is narrow. And "beat" applies to top-1 accuracy only; on referral safety, the chatbots dominated. Treat the experiment as a well-documented anecdote with a genuinely useful lesson, not as a clinical finding.

The lesson itself, though, survives every caveat: the best AI for a job is usually the one built for that job. That's the entire premise of a tools directory — and it's why we keep 1,100+ specialized tools categorized instead of just linking one chatbot.

Frequently Asked Questions