FIELD NOTE 05 / Applied AI
Adding a medical LLM made medication search worse. Here's why.
MedGemma fixed 11 searches and broke 22 that plain search already had right. A frozen benchmark, a held-out check, and a live GPU rerun show where always-on LLM reranking hurts.
MEDICATION RESOLUTION / FIELD NOTE 1 / V0 → V2
Research artifact. Not clinical guidance.
I've worked in healthtech (including a startup) and, more recently, on benchmarking. This note is where the two meet.
It's part of a larger capstone: an evaluation-driven system for resolving medications and their interactions. Input may eventually be typed, spoken, or scanned, and before the system can retrieve interaction evidence it has to turn messy medication mentions into canonical products. I'm building it incrementally: every stage is measured against the same frozen benchmark, and the next architectural decision comes from the previous stage's failures.
This note covers one of those decisions: should a medical LLM sit after retrieval and rerank every candidate set?
I expected the interesting part of this experiment to be whether a small medical model could rescue difficult searches.
Instead, it answered a different question: if retrieval already puts the right medication first, can an LLM reranker actually make the answer worse?
This is an engineering note on a small, frozen benchmark. It covers medication resolution: taking a messy search query and identifying the intended product, ingredients, and strength. It is not clinical advice, and it is not the full capstone.
In the whole capstone, this is step 3:
1. Input
2. Parse / normalise
3. Resolve medication ← this note
4. Confirm product
5. Resolve interactions ← not built yet
6. Ground evidence ← not built yet
7. Generate cited output ← not built yet
And inside step 3 the experiments are:
Resolution experiments
V0 Search DB (Typesense) baseline
V1 deterministic preprocessing
V2 MedGemma rerank-all ← this note
V3 selective LLM invocation
V4 query reinterpretation + second retrieval
The benchmark has 100 products and 603 queries. Of those, 503 are product-name searches (100 clean, 403 with noise) and 100 are ingredient/composition searches.
The metric is deliberately simple:
Did the first result match the intended product?
Everything is frozen and replayable. The numbers on this page come from the recorded benchmark artifacts rather than being typed into the article.
REPLAYABLE EVIDENCE
Reproducing the run
Choose the verified artifact replay or the recorded A5000 replay below. They reached the same answers, but they are different records: one is reconstructed from the frozen results; the other is verbatim worker output from the live rerun.
I reran V2 on an RTX A5000 serverless GPU, the same GPU type used for the original experiment. The model version was pinned and the benchmark hash checked before execution.
Across all 603 queries, the live rerun produced the same candidate choice and the same correct/incorrect outcome as the recorded run.
Average reranking latency was 148 ms per query, against 222 ms in the original run. The software stack had changed; the answers had not.
A cold host can take up to ~3 min to become ready; startup is ~15 s if the image is cached.
V0 VS V2 · REPLAY + LIVE GPU
Does adding a medical LLM get more searches right?
V0 is search alone. V2 is the same search, with MedGemma picking the answer from the top five results on every query. Same 603 searches, same answer key.
Checking live availability…
Final result of the recorded run. Replay it or run it live to watch it build, one search at a time.
What MedGemma changed
MedGemma fixed 11 searches but broke 22 that search alone already had right. Net: 11 fewer right answers than search alone.
Every square is one of the 603 searches. Most never changed; the coloured ones are where MedGemma made a difference. Select one to open it in the explorer.
- MedGemma fixed it 11
- MedGemma broke it 22
- Right both times 484
- Wrong both times 86
No replay selected. “Verified artifact” is reconstructed from frozen results; “recorded A5000 run” is the verbatim 2026-09-27 worker output.
0 / 603 · ✓ 0 · fixed 0 · regressions 0
Why 82.9% in the terminal but 82.1% here? The script dropped 6 unreadable answers from its count (495 / 597). This note counts them as misses (495 / 603), the same way search alone is scored.
Speed, likewise: the terminal's 222 ms average includes 33 searches with nothing to choose from (0 ms). When the model actually ran, it averaged 234.6 ms.
First: how good was plain retrieval?
The V0 baseline is Typesense, a search DB.
It gets 506 of 603 queries right: 83.9% Top-1 accuracy.
Ingredient searches are almost solved at 98%. Product-name searches are harder at 81.1%.
The interesting number is further down the ranking: for product-name queries, the correct answer is somewhere in the Top-5 87.3% of the time.
That leaves an obvious opening.
Retrieval often knows the answer. It just doesn't always put it first.
A few examples show the kinds of failure involved:
eRpra-DSR
→ Zepra-DSR 30mg/40mg
wanted: Repra-DSR
One swapped letter changes the product.
azivay200mgoral
→ no result
Removing spaces can break retrieval entirely.
And:
Azivay 200 Oral
→ Ofzi 200
wanted: Azivay 200mg Oral
The weakest noise category is compact text: only 53 of 99 queries are correct.
So the next question seemed reasonable: if the right product is often already in the Top-5, could a model read those five candidates and choose better? That's what reranking is for.
The number worth carrying from V0 is 87.3%: Search-based retrieval already puts the intended product somewhere in the Top-5 surprisingly often.
Before trying the LLM, I also tried deterministic preprocessing, hoping to avoid the latency and cost of calling a reranker at all. I dropped it without keeping a score: aggressive rules broke valid names. That experiment reinforced the same principle: every extra transformation needs to earn its place.
EXPERIMENT CHRONOLOGY
-
V0
Typesense baseline Measured
-
V1
Deterministic preprocessing Tried, dropped
-
V2
MedGemma rerank-all Field note 1
-
V3
Selective LLM invocation Next: field note 2
-
V4
Query reinterpretation + retrieval Planned
-
V5
Fine-tuning, if justified Planned
So I added MedGemma
MedGemma is Google's collection of open-weight medical AI models built on Gemma 3 for medical text and image comprehension.
V2 of this experiment keeps Typesense as retrieval and adds MedGemma 1.5 4B as a reranker:
query
↓
Typesense Top-5
↓
MedGemma chooses 1–5
↓
answer
The model sees every search result, including the easy ones.
The result: 495 / 603 correct (82.1%), against 506 for Typesense alone. Adding the model lost 11 correct answers overall.
That headline hides both sides of the experiment. MedGemma fixed 11 searches that Typesense got wrong. But it also broke 22: 18 wrong selections and 4 answers that could not be parsed as the requested choice.
It wasn't a quirk of these 603 queries either. On a second set of 600 new queries, built from 100 different products, the same always-on reranking again scored below plain search: 481 against 492.
The useful part is not simply that the score went down. It is where the model helped and where it caused damage.
MEASURED / SAME 603-QUERY DENOMINATOR
Rerank-all lost 11 correct answers.
506 / 603 correct
495 / 603 correct
When the reranker helped
Three examples show where it helped. For rabium40, Typesense ranked
Rabium 20 first; the intended Rabium 40 was in the candidates, and MedGemma
moved it up. For Jeprifa 100 5, it replaced
Jeprifa 200mg with the correct Jeprifa 100mg/5ml. And it recovered
the typo iZdkof-AX as Zidkof-AX.
These are exactly the cases I hoped reranking would help with: the retrieval system has found plausible candidates, but interpreting the whole product name — especially strength and formulation — changes which one should win.
My mental model: search-based retrieval had already put the right set of files on the desk. The medical LLM sometimes helped by opening the five folders, reading the labels more carefully, and moving the right one to the top.
But then look at Clazin
For:
Clazin 80mg
Typesense returned the intended product:
Clazin 80mg
MedGemma changed it to:
Clazin M 80mg/500mg
That wasn't a one-off.
It made the same wrong choice on all five name-style Clazin queries; clean, typo, compact and missing-unit variants included.
Typesense had already got every one of them right.
The same pattern appears elsewhere:
Stator 20
→ Stator Gold 75mg/20mg/75mg
and:
Floxiven 200
→ Floxiven OZ 200mg/500mg
The model is repeatedly attracted to the longer combination product when a simpler product name already matches what the user typed.
Open these regressions in the benchmark explorer →
This is the part I find most useful. The model's fixes and its damage happen in different places, and most of the damage occurs where retrieval was already right. The problem isn't that it can't rerank; it's that I asked it to rerank everything.
There was another failure mode
The prompt asked the model to return a number from 1 to 5.
Sometimes it returned something like:
FINAL: Evoway Plus
That happened 6 times in total. 4 of those were on queries Typesense had already got right, three of them for the same product.
This is mundane compared with the ranking errors, but operationally it matters. A model can understand the task and still violate the interface contract.
That belongs in the result, not hidden as a parsing detail.
Method and numbers in full
Benchmark
Products 100
Queries 603
Product-name queries 503
Composition queries 100
clean 100
typo delete 100
typo swap 100
compact 99
strength without units 62
punctuation 35
form removed 7
composition 100
V0 — Typesense
Top-1: 506 / 603 = 83.9%
Composition R@1: 98.0%
Product-name R@1: 81.1%
Product-name R@5: 87.3%
V2 — Typesense + MedGemma
Top-1: 495 / 603 = 82.1%
Fixed 11
Lost 22
wrong reranker choice 18
unreadable model answer 4
Not attempted 33
One terminal output shows 82.9%. That is 495 / 597, using parsed output rows as the denominator. The comparison in this note uses all 603 benchmark queries.
So what should V3 do differently?
The result doesn't tell me that LLM reranking is a bad idea.
It tells me that reranking every query is a bad policy for this pipeline.
Typesense is already very good on clean and obvious searches. Calling another model there creates a new opportunity to destroy a correct answer.
The next experiment is therefore selective invocation.
Call the model only when retrieval gives us a reason to distrust Top-1.
For example:
query strength ≠ Top-1 strength
or:
Top-1 name poorly resembles the typed name
I've since tested that rule on 600 new queries from 100 products it had never seen. It held up: it called the model far less often, broke nothing, and got more right than search alone. That's the next note.
The goal is no longer:
Can the LLM choose better?
It is:
Can we identify when the LLM has earned the right to choose?
What this note does not claim
This is one small experiment:
- one reranking model,
- one prompt,
- one Typesense Top-5 candidate set,
- 100 products,
- synthetic noise,
- 603 frozen queries.
It is not evidence that LLM reranking generally makes search worse.
It is evidence that this particular always-on reranking policy made this benchmark worse, and the recorded outputs let us inspect exactly how.
V4 will try a different role for the model entirely: instead of choosing from retrieved results, let it rewrite the messy query and retrieve again.