Case Study — Diagnostic Odyssey
Work aimed at patients whose diagnosis takes years — making a fragmented medical journey legible, and finding signal in routine clinical data early enough to justify a second look. Two tracks so far: clinical NLP over records, and pancreatic cancer detection on abdominal CT. Built in public, with each experiment designed to retire a hypothesis before it becomes code.
At a glance
Some patients spend years undiagnosed, accumulating records across specialties that no one reads end to end. The system I wanted to build reconstructs that journey from the raw records, surfaces evidence that was overlooked or contradicted, and shows what would distinguish the remaining possibilities — with every claim traceable to its source. The engineering question was which part of that actually needs building, given how capable frontier models already are.
Everything — experiment design, case construction, the perturbation and extraction protocols, scoring infrastructure, analysis, and the public write-ups. This is my main ongoing project, built in public: every experiment becomes a post in The Differential series.
Ran a single-case pilot on a diagnosis-masked neuromuscular case from the literature. A frontier model ranked the published diagnosis first and predicted the pathological evidence that would have resolved the case.
Converted the structured timeline into an ordinary clinical paragraph — the model's answer barely changed, suggesting input structure alone is not a durable advantage on short vignettes.
Ablated individual supporting findings. Removing the characteristic muscle distribution lowered confidence appropriately; removing the strongest electrodiagnostic evidence did not — the model kept its diagnosis at high confidence and predicted the missing study would show the very pattern that had been removed.
Tested missing-vs-normal: the model correctly described an undocumented exam as undocumented rather than normal, but the distinction never showed up in its ranking.
Injected strong ALS counterevidence. The model switched its top diagnosis appropriately — but kept proposing the muscle biopsy from its original narrative as the most valuable next test.
Rebuilt the protocol around single-finding perturbations of one baseline case — characteristic muscle distribution, objective upper-motor-neuron signs, sensory abnormalities, a bulbar symptom, and a normalized lab value — then ran every condition three times in fresh conversations. Eighteen outputs total.
The leading diagnosis proved highly reproducible. Adding upper-motor-neuron findings reversed the top two diagnoses in all three repetitions, and the model preserved the conflicting myopathic evidence instead of erasing it. The characteristic muscle pattern reproducibly strengthened the myopathy; normalizing the creatine kinase reproducibly changed little.
Everything below the leading diagnosis was not reproducible. In the unchanged baseline, the rank and compatibility score assigned to ALS varied considerably between runs — so small evidence effects couldn't be separated from run-to-run variance, and a one-point score change can't be read as a change in diagnostic belief.
Found that the sensory variation exposed my benchmark rather than the model: it was proposing two coexisting processes, and a single 'compatibility' score conflated 'explains an important component' with 'explains the whole case alone'.
Traced the one output that was almost too stable: muscle biopsy proposed as the most informative missing evidence even where ALS had become the leading diagnosis — leaving open whether next-evidence choice is an information-gain estimate or the completion of a familiar diagnostic story.
Moved one step back in the pipeline: instead of asking whether the model can diagnose a clean vignette, asked whether it can turn unstructured clinical narrative into the evidence structure a longitudinal system would reason over. Evaluated on two public clinical NLP benchmarks, CaseReportBench and E3C.
Ran 50 manual extractions — 40 locked test cases plus 10 development examples — each in a fresh conversation on a fixed prompt version, with prompts, outputs, splits, scorer config, and file hashes recorded in a manifest. Deliberately tedious, so the baseline couldn't be improved after the fact by quietly editing prompts or repairing outputs.
Found the opposite of the expected failure. Across 1,070 CaseReportBench claims and 1,459 E3C entities, exact source-quote grounding was 100% and the unsupported-claim rate was 0%. All 49 incorrect character offsets were off by five characters or fewer, and every quote was still uniquely locatable in the source.
The failure was structural. The model emitted 2.79 grounded claims for every gold claim — fragmenting and merging evidence units rather than inventing them — giving a relaxed span F1 of 0.199. Entity boundaries were similarly unstable: strict span F1 0.140, rising to 0.407 under partial overlap.
Time was weakest. Temporality macro-F1 was 0.216 and mapped-endpoint temporal-relation F1 was 0.031, with only about half of relation endpoints resolvable at all — a temporal graph can't hold when the identities of the two events it connects aren't stable. Two of 40 outputs also failed to parse as JSON, which is fine for a chat window and not fine for a pipeline.
Changed domains for a methodological reason. The text benchmark's poor score was arguable — a missed finding, a paraphrase, or a granularity disagreement all look alike — so I moved to a task where a false positive can't be explained away as phrasing: detecting pancreatic ductal adenocarcinoma on routine contrast-enhanced abdominal CT, as a second-pass signal on imaging that already exists.
Built the pilot on PANORAMA (2,238 portal-venous studies; 676 PDAC, 1,562 non-PDAC) because it supports evaluation at two levels at once — a case label for whether the model separates PDAC from non-PDAC, and a lesion mask for whether it was looking in the right place.
Ran five-fold cross-validation over 168 development cases from a 198-case manifest, holding 30 cases as a locked test set excluded from model selection. Fixed seed, 25 epochs, batch size 1 — volumetric CT is memory-bound, and Colab session limits made a completed run something to preserve rather than casually reproduce.
Pooled out-of-fold AUROC was 0.5244, barely above chance, with 16.7% specificity at 90% sensitivity — not a deployable screening system. But gold-lesion detection sensitivity was 0.7439 against a Dice of only 0.1086, which separates 'sees nothing' from 'notices the right area and represents it badly'.
Found the more important result by not reporting the flattering one. Fold-wise AUROC averaged 0.6109 ± 0.0447, but predictions were compressed into roughly 0.936–0.957 for both classes, so per-fold score scales weren't comparable and the pooled ranking collapsed. Reporting the fold average alone would have hidden that a global threshold has no stable meaning.
Four experiments published as Parts 1–4. Together they retired three hypotheses and produced two failure modes precise enough to act on. Clean-vignette diagnosis is not the wedge: the model reproduces its leading diagnosis and updates correctly on strong evidence. Hallucinated evidence is not the risk either: source grounding was perfect across roughly 2,500 extracted claims — what broke was evidence units, entity boundaries, and temporal relations. On the imaging side, the first pancreatic CT run learned a weak localization signal (0.74 detection sensitivity) but produced patient-level probabilities too compressed to be comparable across folds. Current state: a locked 30-case test set still untouched, and a debugging queue ordered by what would invalidate the pipeline fastest — evaluation audit, visual error groups, aggregation rules, then calibration on held-out predictions only.
The recurring lesson is how often the experiment indicts my own assumptions rather than the model. I expected insensitivity to evidence and found a reproducibility hierarchy instead. I expected hallucination and found flawless citation with unusable structure. Twice the measurement itself was the bug — a compatibility score that silently asked two different questions, and a benchmark F1 that punishes a defensible choice about whether a lab panel is one finding or six. That is what drove the move to imaging: I wanted a task whose failures I couldn't argue with, and switching domains bought a cleaner signal about my own pipeline more than about any model. The discipline I've settled on is choosing experiments by how unambiguously they can fail, reporting the unflattering number when the flattering one exists (fold-average AUROC looked far better than the pooled result that mattered), and refusing to let infrastructure justify itself — if a layer doesn't improve the output a patient or clinician would actually read, it shouldn't exist. This section updates as the project progresses.