Appen MedTerm-90 Benchmark

ElevenLabs Scribe v2 leads at 83.35%. OpenAI gpt-transcribe (81.79%) and Microsoft MAI-Transcribe 2 (81.31%) are a statistical tie for second with different characters: gpt-transcribe spells medicine cleanly, while MAI-2 pairs near-top recall with 94× real-time speed. Google's Gemini 3.5 debuts fourth at 80.65%, within the statistical margin of third, takes two categories outright, and needed the fewest corrections in the field. Muse and Amazon fill the middle, and Grok pairs the field's highest speed with its lowest recall.

Scribe v2 leads three of seven categories (symptom 90.0%, procedure 85.7%, anatomy 84.0%). Gemini 3.5 takes diagnosis (83.7%) and lab values (78.7%). Amazon owns medications at 83.0% while dropping to 51.1% on vital signs, and gpt-transcribe takes vital signs (78.6%). Grok misses roughly one medication in two. Lab values are the weakest category field-wide, because the judge does not forgive unit and decimal garbles.

The ranking moves with the specialty. gpt-transcribe takes psychiatry (82.8%), and Gemini 3.5 edges the medical-legal lead (86.0%). Scribe v2 leads neurology (88.9%) and orthopedics (65.8%). Orthopedics remains the hardest major specialty for every model, with a 23-point spread down to Grok's 43.0%.

Grok is fastest at 102× and least accurate. MAI-Transcribe 2 holds the speed-accuracy frontier: 94× real time at 81.28%. Gemini is second fastest at 44.7×. Muse is slowest (6.5×, 28.9 s median), and Amazon's 8.8× includes its asynchronous upload, queue and poll pipeline, which is the latency production users actually experience. Per-file latency (p50 / p95): Scribe v2: 10.8s / 21.4s · gpt-transcribe: 6.6s / 18.3s · MAI-2: 2.1s / 4.4s · Gemini 3.5: 4.0s / 12.2s · Muse: 28.9s / 86.5s · Amazon Med.: 21.4s / 53.4s · Grok 1.0: 1.9s / 4.3s.

On 28.0% of files no model reached 85% recall, and 192 files defeated all seven. The safety chart counts files where a model missed over half the medications or vital signs: Scribe v2 (20.4%) and MAI-2 (20.7%) fail that way least, and Grok does on more than half its files.

Recall percentages treat every error the same. Clinically they are not the same. A dropped decimal turns Eliquis 2.5 mg into a tenfold anticoagulant overdose. A one-syllable slip turns Toradol, an NSAID, into tramadol, an opioid. These misrenderings look plausible on the page, they survive casual review, and several were written by five different models on the same file. Our risk judge classified every near-miss pair: 191 were rated potentially fatal and 1,027 serious. Every engine in the field produced some.

Each pair was rated on a five-level scale. Fatal: acting on the transcript could directly cause life-threatening harm, such as a wrong drug class, an order-of-magnitude dose, or a tripled frequency. Serious: significant clinical harm or a major treatment error is plausible, such as a similar but wrong drug or a lost critical order. Moderate: could mislead care but would likely be caught in context. Minor: a cosmetic deviation a reader recovers instantly. None: clinically equivalent. Across all 6,425 pairs: 191 fatal, 1,027 serious, 1,566 moderate, 2,027 minor, 1,614 none.

The drug-name garble gallery

516 of the fatal and serious renderings are outright drug-name garbles: one medication becoming another, or becoming a non-word. 444 of those were made by a single model; no other engine made the same mistake on the same audio. A sample of those single-model errors:

Sound-alike drug pairs are a known hazard in human transcription, and each model has its own: a cardiac drip becomes an opioid, an expectorant becomes an antibiotic, an NSAID becomes Xanax. No other engine made the same mistake on the same audio, which makes these model-specific confusions rather than audio problems. Each swap is a different pharmacology delivered to the chart in confident prose.

The corpus: real data, zero contamination

1,413 scored files (of 1,424 collected): real physician dictations with paired scribe-authored SOAP notes. All of it is dictation audio: a single physician speaking close to the microphone, no crosstalk, no far-field capture. That makes the conditions favorable, and the remaining errors harder to excuse. 90.5 hours, median 3 min 05 s, 14 specialties led by medical-legal/IME (36%), neurology, surgery, orthopedics and psychiatry. The audio is dominated by telephony: 1,003 files (70%) are GSM 6.10-compressed WAV from telephone dictation lines, which carry codec-limited, telephony-band speech. The rest splits between 235 MP3 recordings (44.1 kHz mono, 320 kbps) and 186 uncompressed PCM WAV files (16 kHz). Estimated SNR by the frame-energy percentile method: PCM median 24 dB, MP3 33 dB, GSM 44 dB. The high GSM figure reflects codec noise-squelching, not better speech. Every recording is real data from real clinical environments. Nothing is synthetic, scripted or studio-read. The corpus is part of Appen's licensed dataset library, and only audio never sold to or acquired by any of the companies tested was used, so no model could have seen this data in training.

How each API was called

Every model ran through its standard production interface with vendor-recommended defaults. Runs were checkpointed per file, failures logged and retried, and latency measured as the full API round-trip. Expand a model for the exact request and response shape:

POST https://api.elevenlabs.io/v1/speech-to-text
xi-api-key: ****
multipart/form-data:
  model_id = "scribe_v2"       language_code = "en"
  diarize = "true"             timestamps_granularity = "word"
  tag_audio_events = "false"   file = <raw audio bytes>

{"text": "... blood pressure one twenty-six over sixty-two ...", "words": [...], "language_code": "eng"}

POST https://api.openai.com/v1/audio/transcriptions
Authorization: Bearer ****
multipart/form-data:
  model = "gpt-transcribe"
  file  = <raw audio bytes>      (language auto-detected)

{"text": "... medication like metformin, omeprazole, ... ...", "languages": ["en"], "usage": {...}}

POST {AZURE_ENDPOINT}/speechtotext/transcriptions:transcribe?api-version=2025-10-15
Ocp-Apim-Subscription-Key: ****
multipart/form-data:
  definition = {"enhancedMode": {"enabled": true, "model": "mai-transcribe-2",
                "transcribeStyle": "verbatim"}, "locales": ["en-US"]}
  audio = <raw audio bytes>

{"combinedPhrases": [{"text": "... on medication metformin, omeprazole, Invokana, Unglyza, glipizide ..."}]}

google-genai SDK -> models.generate_content
model = "gemini-3.5-transcribe"
contents = [<audio bytes inline>, "Transcribe this medical dictation verbatim."]
(one persistently empty response file excluded from the common subset for all
 models; see limitations)

{"candidates": [{"content": {"parts": [{"text": "... dictating a monthly follow-up ..."}]}}]}

POST https://api.meta.ai/v1/asr/transcribe?sessionId=<uuid>
Authorization: Bearer ****
multipart/form-data:
  request = {"mode": "DIARIZATION", "model": "muse-voice-transcribe-1.0",
             "audioEncoding": "WAV"}
  audio = <16 kHz mono PCM WAV bytes>   (files > 580 s chunked)

{"text": "... blood pressure 126 over 62 respirations 18 pulse 74 ...", "segments": [...]}

boto3 (us-east-1): upload to S3, then
StartMedicalTranscriptionJob(
  LanguageCode="en-US", Specialty="PRIMARYCARE", Type="DICTATION",
  MediaFormat="mp3"|"wav", Media={"MediaFileUri": "s3://<bucket>/<key>"})
poll GetMedicalTranscriptionJob every 5 s -> fetch result JSON from S3

{"results": {"transcripts": [{"transcript": "... on medication metformin, omeprazole, Invokana. Onglyza ..."}]}}

POST https://api.x.ai/v1/stt
Authorization: Bearer ****
multipart/form-data:
  format = "true"    language = "en"
  file = <raw audio bytes>    (NO keyterm parameters; GSM WAV pre-converted to PCM)

{"text": "... blood pressure 126 over 62, respirations 18, pulse 74. ...", "words": [...]}

Scoring methodology

Reference entities were extracted from the scribe notes at temperature 0 and reviewed by a cohort of medical experts, who also spot-checked the final outputs and scores of every model. To score a transcript, entity and transcript first pass through the same clinical normalization: casing and punctuation are unified and spoken numbers canonicalized, so that “126/62”, “126 over 62” and “one twenty-six over sixty-two” are the same blood pressure. Matching then runs exact, fuzzy (indel ≥ 80), and bidirectional abbreviation, in that order.

Character similarity finds near-misses, but it cannot tell a formatting difference from a misspelling: “para vertebral” and “Unglyza” look equally close to their references. So every one of the 12,326 distinct near-miss pairs was reviewed the way a clinician would read it (Gemini 3.8 Flash, temperature 0, fixed policy). A rendering is credited when the written form is one clinicians actually use: spacing and hyphenation differences, recognized alternate spellings, standard abbreviations, equivalent number formats, minor inflections. This is what protects models from being punished for style — a model that writes “b.i.d.” where the note says “twice daily”, or hyphenates a compound term, loses nothing. What gets rejected is substance: misspellings and non-words (“Unglyza” for the drug “Onglyza”), different drugs or structures however similar, and garbles a clinician would flag. 21.2% of pairs were accepted; rejected credits count as misses. Canary checks passed: misspelled brand names rejected, spacing-only variants accepted.

Neutrality is verified rather than assumed. In decoy testing, identical clinical entities from unrelated files produced statistically indistinguishable false-match rates across models (7.6–8.4%), so no model gains an advantage from verbosity or output style. Every model faced identical files, entities, thresholds and judges, with no vocabulary hints, and every recall figure carries a file-level bootstrap 95% CI (2,000 resamples).

Worked example: one dictation, all seven models

Pulmonology follow-up dictated over a telephone line (GSM 6.10 WAV, 21 reference entities; internal id …163350). The scribe note lists the patient's medications, including “gabapentin 300 mg at bedtime … Incruse inhaler daily.” Each model's rendering of that passage, verbatim:

Scoring of eight representative entities. ✓ credited · ✓f credited via accepted fuzzy match · ✗j near-miss rejected by the judge · ✗ missed:

One inhaler, four fates: Amazon, MAI-2 and gpt-transcribe write “Incruse” and get credit. Grok writes “in-cruz” and Muse writes “incruise”, and the judge rejects both as non-words (✗j). Scribe v2 writes “Qvar”, a different inhaler entirely, and misses. Gemini writes “Incruse Ellipta daily”, which no longer matches the reference phrase and scores as a miss. “Oropharynx” defeats all seven on this recording. Full per-file results and raw API responses are available on request.

Limitations, read before quoting

• Recall only: hallucinated extra entities are not penalized (entity-F1 planned).
• The strict standard is a policy choice. It demands clinician-acceptable written forms; pre-adjudication scores are retained in the benchmark's audit trail.
• The LLM judge is imperfect. Spot-checks show a small false-reject band on benign spacing variants, roughly 20% of spacing-only pairs. It is applied identically to every model and the verdict cache is auditable.
• One file was excluded from the common subset for all models: Google Gemini 3.5 Transcribe persistently returns an empty response for it across reruns. Scoring it would have been a coverage failure, not an acoustic one.
• The reference is the scribe's note, not a verbatim transcript; residual reference imperfections cap achievable recall for all models equally.
• The corpus over-represents medical-legal dictation; small specialties (n<20) are not reported.
• One run per model, a September 2026 snapshot. SNR figures are frame-energy estimates, not calibrated.

About Appen

Appen builds expert data for the frontier of AI: the specialist, hard-to-reach data that generic web corpora cannot supply. The audio behind MedTerm-90 is a case in point. These are real dictations by real practicing physicians, captured in live clinical transcription workflows and licensed for AI development through Appen's data partnerships. Speech like this cannot be scripted, synthesized, or scraped. That access runs deep: licensed real-world archives across regulated domains, a global network of vetted specialists (clinicians among them) for expert annotation and validation, and coverage across 500+ locales and acoustic conditions. The same machinery powering this benchmark is available to model builders: expert domain corpora from Appen's data library, custom collection with specialist ground truth, and evaluation pipelines like this one run on your own models, your own data, your own metrics.