Blog / On-device AI
My fine-tuned search model scored better. I'm not using it yet.
I'm building a voice-notes app for the iPhone where everything stays on the phone: the recordings, the transcripts and the search. Your notes never leave it. That means search has to run on an AI model small enough to live inside the app (about 150 MB), so I wanted to see how much better I could make that small search model by training it further. More on the app itself soon.
The short version:
- What I did. I fine-tuned (retrained) a small search model so that short queries find long voice-memo notes better. It runs entirely on the phone.
- What improved. If search always shows five results, the right note is among them 54.7% of the time, up from 46.0%.
- Why it didn't matter. The app hides results when the model isn't confident, so a wrong note never sits at the top. Counting only what would appear on screen, the fine-tuned model is ahead by less than one point (36.0% vs 35.4%), which is one or two queries.
- Why the improvement vanished. The fine-tuned model got better at putting the right note first. But the app also has to spot a search for something you never recorded, so it can show nothing instead of a wrong note. The fine-tuned model was barely better at that, and that is what decides whether results appear on screen.
- What I got wrong. I picked the checkpoint that ranked best. A later one is much better at spotting searches with no answer. I will score it once, on a human-written test set, before deciding again.
- Outcome. The app keeps the base (original) model for now.
Setup
The app records voice memos on a phone. The notes are long, rambling transcripts, and the searches are about three words.
The model is IBM granite-embedding-english-r2 (149M parameters, ModernBERT, CLS pooling). It turns text into numbers that stand for its meaning, so search can match by meaning and not only by exact words. On the phone it is a compressed (int8) version of about 150 MB that reads 256 tokens at a time.
Search works in two steps. Exact-word matches come first. Everything else is ranked by cosine similarity (a score for how close in meaning the query and a note are). A long note is split into chunks of up to 256 tokens and scored by its best-matching chunk.
The training data was all invented by a script. It has 4,000 notes (8 to 291 words) and 8,061 query-to-note pairs (a short search matched with the one note it should find), with queries averaging 3.3 words. The queries are a mix of keyword, paraphrase, buried-detail and vague ones. The wrong answers the model practises against, called hard negatives, are same-topic notes, same-person notes and contentless filler.
Training used an in-batch contrastive loss (scale 20) with other correct notes masked out: batch 32, learning rate 2e-5, 2 epochs, 502 steps, about 2.5 hours.
Hardware
Training ran on a MacBook Air with an Apple M4 chip (10 cores: 4 performance, 6 efficiency) and 24 GB of memory, using PyTorch on the MPS backend (Metal Performance Shaders, which runs the training on the Mac's built-in GPU). The full fine-tune ran there. The phone is an iPhone 18 Pro Max on iOS 27, the one physical phone the app is tested on.
Fitting the training into 24 GB of memory
The model learns in practice rounds. In each round it gets 32 searches and 32 notes, and has to match every search to its own note. The other 31 notes are the wrong answers it must learn to reject. That group of 32 is called a batch.
The problem is that all 32 have to fit in memory at the same time. The first run peaked at 21.7 GB at step 0 and froze the Mac.
The usual fix is gradient accumulation, which runs four small rounds of 8 and adds up what the model learned from each. That does not work here, because a round of 8 has only 7 wrong notes to reject, so four rounds of 8 are an easier exercise than one round of 32.
What does work is a gradient cache. The model first works through all 32 quickly, without keeping the working it would need to learn from them, and scores how well each search matched its note. Then it goes back through the same 32 in chunks of 8, this time keeping its working, and learns from each chunk using those scores. Only 8 are in memory at a time, but the model still learns from all 31 wrong notes.
In PyTorch terms: embed the whole batch without autograd, take the gradient of the loss with respect to the embeddings only, then re-encode in chunks of 8 and push each chunk's share back. The result is the same gradient as the full batch, as long as the second pass repeats the first one's dropout. The method is GradCache (Gao et al., 2021); sentence-transformers ships it as CachedMultipleNegativesRankingLoss.
Peak memory dropped from 21.7 GB to about 6.5 GB. As a safety net I also set PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.5, so a bad run fails with an error instead of freezing the machine.
Ranking improved
During training the model is saved at regular points, and each saved copy is called a checkpoint. I picked the checkpoint that most often put the right note in the top five on the dev set (queries kept aside for making choices like this). That was step 100.
I tested it on 270 held-out queries the model had never seen. Both models ran as the compressed int8 versions that run on the phone (each number stored in 8 bits instead of 32). There was no cutoff in this test. The search always shows its best matches, however weak they are.
The fine-tuned model was ahead on every slice. Vague queries and very long notes (160+ words) improved the most.
Two things did not change. Exact keywords were already at 86% and stayed there. And queries about a time ("last Tuesday") stayed near zero for both models. An embedding matches notes by meaning and has no calendar, so it cannot know which day "last Tuesday" was. Those queries need date parsing instead. Work out the actual date, then filter the notes by when they were recorded.
In the real app, the fine-tuned model was no better
The test above always shows five results, even when none of them is any good. The app does not work like that. When the search is not sure, it shows nothing, because a wrong note at the top teaches you not to trust search.
Here is an example. Say you search for "dentist appointment", but you never recorded a note about the dentist. The right result is an empty screen. A search model still gives every note a score, so the app has to tell from those scores that none of them is a real match, and show nothing instead of the five least-bad notes. That is a different skill from ranking. Ranking means putting the note first when it exists. This means noticing when it does not exist.
So the search needs a rule for when to show results. The simplest rule is a fixed cutoff, which only shows a note if its score is above one fixed number. I set that number just above the highest score reached by any query that has no matching note, so those queries show nothing. With that rule, the right note was shown in the top five for 29.4% of queries with the base model and 33.1% with the fine-tuned one.
Fine-tuning also shifted every score down. The cutoff that works moved from about 0.85 to about 0.68, so reusing the old cutoff would have hidden most of what the new model ranks correctly. A lower cutoff is neither good nor bad in itself. It only means the cutoff has to be set again for each model.
The app uses a smarter rule that decides for each query separately, and shows results only when the best note clearly stands out. Its score must be at least 0.82, and at least 0.085 above the typical (median) score across all notes for that query. Then the app shows the notes that score within 0.15 of the best one, 10 at most.
Those numbers come from a grid search (trying many combinations and keeping the best) on the dev split, a set of queries kept aside for tuning settings and never used in the final test. The grid search runs under a leak budget, which is how many queries that have no matching note are still allowed to show a result. A budget of 0 means none may.
For a fair comparison I worked out those numbers again for each model separately, so each model got its own best version of the rule. Then I counted how often the app would have shown the right note in its top five.
All 270 test searches have a matching note. The no-match searches were used to set the rule on the dev split, so these scores show what each rule costs on searches that do have an answer.
The result was the same within noise: 35.4% against 36.0% at leak budget 0, and 39.7% against 39.8% at leak budget 1. That is one or two queries out of 270.
To be clear about what I measured, this is the app's rule applied to the 270 test queries, not a trial inside the app. A side-by-side trial on the phone is still to come.
Two things happened. The per-query rule helps the base model more than the fixed cutoff did. And once each model has its own best rule, the 8.7-point improvement from the ranking test does not reach the screen.
Why
The model ranks better but does not separate better by enough. A show-or-hide rule depends on the gap between queries that have an answer and queries that do not.
I picked the checkpoint on the wrong measure. Step 100 was best at ranking. On the dev set, later checkpoints rank slightly worse and separate much better. The match vs no-match gap is 0.107 at step 100 and 0.145 at step 300.
Right note in top 5 (higher is better)
Gap between answer and no-answer scores (higher is better)
Highest score on a search with no answer (lower is better)
I have not scored a later checkpoint on the test split, and I will not keep re-running it until something looks good.
The test flatters the model anyway. It came from the same generator as the training data, so part of the improvement is the model learning the generator.
Next
The next round starts with evaluation, not training. First comes a pass mark, written down before any result. Then a better test set: larger, human-written, and labelled before any model runs, including queries that have no matching note.
With that in place, I will score the step 300 checkpoint once. It already separates better, and it costs no new training. If that is not enough, the next training run will include queries with no matching note, so the model learns when to show nothing.
Time queries like "last Tuesday" get date parsing in ordinary code, since no model will solve those.
Two more things round it off. One is a ceiling, a model trained locally on real transcripts that never ships. The other is a trial on the phone, a blind interleaved comparison where results from both models are mixed together without saying which is which.
Takeaways
- A gradient cache makes large contrastive batches fit on a laptop without changing the loss.
- A top-five score with no cutoff only measures ranking. If you hide uncertain results, measure what is displayed and pick checkpoints by that.
- Calibrate every threshold per model, and compare models at their own calibrated settings.
- A test set from the same generator as the training data can flatter you.