AI Medical Scribes Are Getting Drug Names and Diagnoses Wrong — NHS Watchdog Warning Explained

What Happened: One Dropped Word, One False Diagnosis

The most important AI safety story of the week isn't about frontier models or autonomous agents — it's about transcription. On August 31, Healthwatch England, the statutory NHS patient watchdog, warned that AI scribes — the tools that listen to doctor–patient consultations and automatically write clinical notes — are putting wrong drug names and wrong diagnoses into live medical records, and that in most documented cases the errors were caught by patients, not doctors.

The flagship case is chilling in its simplicity. A woman was informed, based on an AI-generated summary of her hospital consultation, that her MRI scan showed demyelination — serious nerve damage that can underlie multiple sclerosis. The scan had shown no such thing. The actual result read “null demyelination.” The AI scribe dropped one word, and that single dropped word reversed the meaning of a diagnosis. The patient — who happens to be an NHS health professional — only discovered the error because she checked her own record. “It was a very traumatising experience to be given an incorrect diagnosis because of AI and then be told it’s a typo,” she told Healthwatch.

GPs and hospital doctors in England are already using 27 different AI scribe products, and the NHS is rolling the technology out rapidly to cut documentation workload. The warning lands at the exact moment adoption is accelerating — which is precisely what makes it matter for anyone evaluating AI tools for high-stakes work.

The Errors Healthwatch Found

Three patterns recur across the cases Healthwatch England documented:

• Look-alike drug confusion. In one case, an AI scribe recorded the wrong drug entirely — confusing the medication the GP had actually prescribed with a different one with a similar name. In medication records, a near-miss name is not a near-miss outcome.
• Meaning-reversing omissions. The “null demyelination” case: dropping a single qualifier flipped a clean scan into a life-altering finding.
• Hallucinated instructions. Dr Shier Ziser Dawood, a London GP, previously documented a scribe that recorded her telling a patient to “continue their Prozac” — a drug she had never prescribed or discussed. Another AI-generated summary letter omitted a consultant’s instruction to seek a repeat migraine prescription from the GP, which could have left the patient unable to get medication at all.

The common thread in nearly every case: the patient caught it, not the clinician. The humans who were supposed to be the safety net missed the errors that laypeople spotted reading their own records. That detail should reframe how any organization thinks about “human in the loop” as a deployment safeguard — a check that doesn’t reliably check is not a control, it’s a liability allocation strategy.

The Regulation Gap: Why AI Scribes Aren't Medical Devices

Here’s the part that turns isolated errors into a systemic problem. Healthwatch England said it is “worrying” that the Medicines and Healthcare products Regulatory Agency (MHRA) has decided not to classify AI scribes as medical devices — meaning there is no England-wide oversight regime testing these products for safety and effectiveness before or during deployment.

The regulatory line, as experts quoted in the coverage explain, works like this: a system that only transcribes what was said is not a device, while a system that suggests a diagnosis or treatment may well be. That puts enormous weight on how each product describes itself — and creates an obvious incentive. A scribe marketed as a “passive transcriber” avoids the regulatory process that a scribe marketed as a “clinical assistant” would have to pass, even if the underlying technology is similar. The MHRA’s position is that clinicians remain responsible for reviewing and verifying AI-generated transcripts before they’re used in care. The machine writes, the clinician checks, the institution carries the risk — thin comfort when the human check is already missing dropped words, wrong drugs, and hallucinated prescriptions.

Ministers have separately been warned that the NHS and individual medics could face lawsuits over AI-made mistakes. With 27 products in live use and no national oversight, the question isn’t whether errors enter records — it’s how many are never caught.

This Isn't the Failure Mode You Expect

The most instructive detail for the broader AI tools market is what these failures are not. As The Next Web’s analysis put it: nobody here was misled by a hallucinated fact in the classic sense — a real sentence was rendered slightly wrong. And slightly wrong is sufficient when the sentence names a drug, or contains the word “null.”

Most AI safety work, vendor marketing, and buyer due diligence focuses on dramatic failures: fabricated citations, jailbreaks, rogue agents. But in production, the dominant risk of transcription-style AI tools is mundane distortion at the point of highest leverage. Summarization — the feature almost every AI tool now ships — is lossy compression, and what gets lost is rarely random. Negations, qualifiers, and look-alike proper nouns are exactly the tokens models fumble and exactly the tokens that carry medical (or legal, or financial) meaning.

The lesson generalizes well beyond medicine: any AI tool that writes into a system of record — medical charts, CRMs, legal files, financial ledgers — inherits this risk profile. The evaluation question isn’t “is the output impressive?” but “what is the worst possible consequence of the most probable small error?”

How to Choose an AI Scribe That Won't Burn You

None of this means AI scribes are a bad idea — documentation burden is a genuine crisis, and ambient scribes demonstrably return hours to clinicians. It means the buying criteria need to change. Whether you're a practice manager, a health-tech buyer, or evaluating any AI note-taking tool for high-stakes use:

• Interrogate the marketing label. Is the vendor selling “transcription” partly to stay outside medical-device rules? Ask directly what the tool does and doesn't do clinically, in writing.
• Ask for error-rate evidence on negation and drug names. A vendor that can’t quote you measured performance on look-alike drug pairs and negated findings hasn’t measured it.
• Prefer verbatim plus summary, not summary alone. A dropped word in a summary with no traceable source transcript is uncorrectable. Full transcripts make errors findable.
• Design the check for the failure you have. If clinicians review in the 30 seconds between patients, you don’t have human oversight — you have human sign-off. Build review time into workflows and audit a sample of records against audio.
• Give patients access to their records. In every documented case, the patient was the control that worked. Read access is a safety feature, not a courtesy.
• Check the exit and audit trail. Errors discovered months later need provenance: which tool wrote this note, from which audio, when, and how do you correct downstream copies?

AI Medical Documentation Tools Compared

Whichever you pick, the Healthwatch report reframes the purchase: the product is not the note — the product is the note plus the error-detection pathway around it.

Frequently Asked Questions