FIELD NOTE 05 / Applied AI

Adding a medical LLM made medication search worse. Here's why.

MedGemma fixed 11 searches and broke 22 that plain search already had right. A frozen benchmark, a held-out check, and a live GPU rerun show where always-on LLM reranking hurts.

MEDICATION RESOLUTION / FIELD NOTE 1 / V0 → V2

MeasuredReplayLive GPU

Research artifact. Not clinical guidance.

I've worked in healthtech (including a startup) and, more recently, on benchmarking. This note is where the two meet.

It's part of a larger capstone: an evaluation-driven system for resolving medications and their interactions. Input may eventually be typed, spoken, or scanned, and before the system can retrieve interaction evidence it has to turn messy medication mentions into canonical products. I'm building it incrementally: every stage is measured against the same frozen benchmark, and the next architectural decision comes from the previous stage's failures.

This note covers one of those decisions: should a medical LLM sit after retrieval and rerank every candidate set?

I expected the interesting part of this experiment to be whether a small medical model could rescue difficult searches.

Instead, it answered a different question: if retrieval already puts the right medication first, can an LLM reranker actually make the answer worse?


This is an engineering note on a small, frozen benchmark. It covers medication resolution: taking a messy search query and identifying the intended product, ingredients, and strength. It is not clinical advice, and it is not the full capstone.

In the whole capstone, this is step 3:

1. Input
2. Parse / normalise
3. Resolve medication        ← this note
4. Confirm product
5. Resolve interactions      ← not built yet
6. Ground evidence           ← not built yet
7. Generate cited output     ← not built yet

And inside step 3 the experiments are:

Resolution experiments

V0  Search DB (Typesense) baseline
V1  deterministic preprocessing
V2  MedGemma rerank-all        ← this note
V3  selective LLM invocation
V4  query reinterpretation + second retrieval

The benchmark has 100 products and 603 queries. Of those, 503 are product-name searches (100 clean, 403 with noise) and 100 are ingredient/composition searches.

The metric is deliberately simple:

Did the first result match the intended product?

Everything is frozen and replayable. The numbers on this page come from the recorded benchmark artifacts rather than being typed into the article.

Reproducing the run

Choose the verified artifact replay or the recorded A5000 replay below. They reached the same answers, but they are different records: one is reconstructed from the frozen results; the other is verbatim worker output from the live rerun.

I reran V2 on an RTX A5000 serverless GPU, the same GPU type used for the original experiment. The model version was pinned and the benchmark hash checked before execution.

Across all 603 queries, the live rerun produced the same candidate choice and the same correct/incorrect outcome as the recorded run.

Average reranking latency was 148 ms per query, against 222 ms in the original run. The software stack had changed; the answers had not.

A cold host can take up to ~3 min to become ready; startup is ~15 s if the image is cached.

Does adding a medical LLM get more searches right?

V0 is search alone. V2 is the same search, with MedGemma picking the answer from the top five results on every query. Same 603 searches, same answer key.

Checking live availability…

Final result of the recorded run. Replay it or run it live to watch it build, one search at a time.

V0 Search alone 506 / 603 right
V2 + MedGemma on every query 495 / 603 right −11 vs V0

What MedGemma changed

Fixed 11
Broke 22

MedGemma fixed 11 searches but broke 22 that search alone already had right. Net: 11 fewer right answers than search alone.

Every square is one of the 603 searches. Most never changed; the coloured ones are where MedGemma made a difference. Select one to open it in the explorer.

  • MedGemma fixed it 11
  • MedGemma broke it 22
  • Right both times 484
  • Wrong both times 86

No replay selected. “Verified artifact” is reconstructed from frozen results; “recorded A5000 run” is the verbatim 2026-09-27 worker output.

Start either replay to enable Pause and Skip to results.

Live GPU is separate from replay. It stays disabled until the deployed live service and its safety flag are enabled.

Live warm-up: up to ~3 min on a cold host; ~15 s if cached. A full run takes about 4.5 min once warm.

Why 82.9% in the terminal but 82.1% here? The script dropped 6 unreadable answers from its count (495 / 597). This note counts them as misses (495 / 603), the same way search alone is scored.

Speed, likewise: the terminal's 222 ms average includes 33 searches with nothing to choose from (0 ms). When the model actually ran, it averaged 234.6 ms.

First: how good was plain retrieval?

The V0 baseline is Typesense, a search DB.

It gets 506 of 603 queries right: 83.9% Top-1 accuracy.

Ingredient searches are almost solved at 98%. Product-name searches are harder at 81.1%.

The interesting number is further down the ranking: for product-name queries, the correct answer is somewhere in the Top-5 87.3% of the time.

That leaves an obvious opening.

Retrieval often knows the answer. It just doesn't always put it first.

A few examples show the kinds of failure involved:

eRpra-DSR
→ Zepra-DSR 30mg/40mg
wanted: Repra-DSR

One swapped letter changes the product.

azivay200mgoral
→ no result

Removing spaces can break retrieval entirely.

And:

Azivay 200 Oral
→ Ofzi 200
wanted: Azivay 200mg Oral

The weakest noise category is compact text: only 53 of 99 queries are correct.

So the next question seemed reasonable: if the right product is often already in the Top-5, could a model read those five candidates and choose better? That's what reranking is for.

The number worth carrying from V0 is 87.3%: Search-based retrieval already puts the intended product somewhere in the Top-5 surprisingly often.

Before trying the LLM, I also tried deterministic preprocessing, hoping to avoid the latency and cost of calling a reranker at all. I dropped it without keeping a score: aggressive rules broke valid names. That experiment reinforced the same principle: every extra transformation needs to earn its place.

  1. V0
    Typesense baseline Measured
  2. V1
    Deterministic preprocessing Tried, dropped
  3. V2
    MedGemma rerank-all Field note 1
  4. V3
    Selective LLM invocation Next: field note 2
  5. V4
    Query reinterpretation + retrieval Planned
  6. V5
    Fine-tuning, if justified Planned

So I added MedGemma

MedGemma is Google's collection of open-weight medical AI models built on Gemma 3 for medical text and image comprehension.

V2 of this experiment keeps Typesense as retrieval and adds MedGemma 1.5 4B as a reranker:

query
  ↓
Typesense Top-5
  ↓
MedGemma chooses 1–5
  ↓
answer

The model sees every search result, including the easy ones.

The result: 495 / 603 correct (82.1%), against 506 for Typesense alone. Adding the model lost 11 correct answers overall.

That headline hides both sides of the experiment. MedGemma fixed 11 searches that Typesense got wrong. But it also broke 22: 18 wrong selections and 4 answers that could not be parsed as the requested choice.

It wasn't a quirk of these 603 queries either. On a second set of 600 new queries, built from 100 different products, the same always-on reranking again scored below plain search: 481 against 492.

The useful part is not simply that the score went down. It is where the model helped and where it caused damage.

Rerank-all lost 11 correct answers.

Inspect all 603 rows →
V0 / TYPESENSE 83.9%

506 / 603 correct

V2 / + MEDGEMMA 82.1%

495 / 603 correct

11fixed 22lost in total
18wrong choice
4lost to parse errors

Held out: on 600 new queries from 100 different products, V0 scored 492; V2 scored 481 (13 fixes, 24 regressions).

Accuracy only. That run used an RTX 3090, so its latency is not compared with the A5000 run.

When the reranker helped

Three examples show where it helped. For rabium40, Typesense ranked Rabium 20 first; the intended Rabium 40 was in the candidates, and MedGemma moved it up. For Jeprifa 100 5, it replaced Jeprifa 200mg with the correct Jeprifa 100mg/5ml. And it recovered the typo iZdkof-AX as Zidkof-AX.

These are exactly the cases I hoped reranking would help with: the retrieval system has found plausible candidates, but interpreting the whole product name — especially strength and formulation — changes which one should win.

My mental model: search-based retrieval had already put the right set of files on the desk. The medical LLM sometimes helped by opening the five folders, reading the labels more carefully, and moving the right one to the top.

But then look at Clazin

For:

Clazin 80mg

Typesense returned the intended product:

Clazin 80mg

MedGemma changed it to:

Clazin M 80mg/500mg

That wasn't a one-off.

It made the same wrong choice on all five name-style Clazin queries; clean, typo, compact and missing-unit variants included.

Typesense had already got every one of them right.

The same pattern appears elsewhere:

Stator 20
→ Stator Gold 75mg/20mg/75mg

and:

Floxiven 200
→ Floxiven OZ 200mg/500mg

The model is repeatedly attracted to the longer combination product when a simpler product name already matches what the user typed.

This is the part I find most useful. The model's fixes and its damage happen in different places, and most of the damage occurs where retrieval was already right. The problem isn't that it can't rerank; it's that I asked it to rerank everything.

There was another failure mode

The prompt asked the model to return a number from 1 to 5.

Sometimes it returned something like:

FINAL: Evoway Plus

That happened 6 times in total. 4 of those were on queries Typesense had already got right, three of them for the same product.

This is mundane compared with the ranking errors, but operationally it matters. A model can understand the task and still violate the interface contract.

That belongs in the result, not hidden as a parsing detail.

Method and numbers in full

Benchmark

Products                         100
Queries                          603
Product-name queries             503
Composition queries              100

clean                            100
typo delete                      100
typo swap                        100
compact                           99
strength without units            62
punctuation                       35
form removed                       7
composition                      100

V0 — Typesense

Top-1:       506 / 603 = 83.9%
Composition R@1:         98.0%
Product-name R@1:        81.1%
Product-name R@5:        87.3%

V2 — Typesense + MedGemma

Top-1:       495 / 603 = 82.1%

Fixed                           11
Lost                            22
  wrong reranker choice         18
  unreadable model answer        4
Not attempted                   33

One terminal output shows 82.9%. That is 495 / 597, using parsed output rows as the denominator. The comparison in this note uses all 603 benchmark queries.

So what should V3 do differently?

The result doesn't tell me that LLM reranking is a bad idea.

It tells me that reranking every query is a bad policy for this pipeline.

Typesense is already very good on clean and obvious searches. Calling another model there creates a new opportunity to destroy a correct answer.

The next experiment is therefore selective invocation.

Call the model only when retrieval gives us a reason to distrust Top-1.

For example:

query strength ≠ Top-1 strength

or:

Top-1 name poorly resembles the typed name

I've since tested that rule on 600 new queries from 100 products it had never seen. It held up: it called the model far less often, broke nothing, and got more right than search alone. That's the next note.

The goal is no longer:

Can the LLM choose better?

It is:

Can we identify when the LLM has earned the right to choose?
What this note does not claim

This is one small experiment:

  • one reranking model,
  • one prompt,
  • one Typesense Top-5 candidate set,
  • 100 products,
  • synthetic noise,
  • 603 frozen queries.

It is not evidence that LLM reranking generally makes search worse.

It is evidence that this particular always-on reranking policy made this benchmark worse, and the recorded outputs let us inspect exactly how.

V4 will try a different role for the model entirely: instead of choosing from retrieved results, let it rewrite the messy query and retrieve again.