# Small Models in Production: What 100+ Blinded Duels Taught a One-Person Travel App

> How Audiala routes five model families by workload — and what happened when I peer-reviewed my own eval and fact-checked 406 'hallucinations' against the web.

URL: https://audiala.com/blog/small-models-in-production/

Calvin French-Owen's "Small Models Have Arrived" made the rounds on Hacker News this week. His thesis — small, cheap models have crossed the threshold where real products become economically viable — matched something I've been living for the past year. I run Audiala, an AI audio travel guide, alone. Solo founder, no infra team, and an LLM bill that has to fit inside a bootstrapped budget. That constraint forced me to do what the big labs' pricing pages quietly discourage: measure everything, and route every workload to the cheapest model that passes its own eval.
When I first wrote up the results, my headline finding was that the most expensive frontier models "fabricated the most." Then I did what I'd want a peer reviewer to do: I went back to the raw eval records, pulled every claim my judges had flagged as a hallucination, and fact-checked all 406 of them against the web. That changed half the story — in a way that's more useful than the original. This post has the corrected numbers, the raw failure examples, and what actually survives.
The setup Audiala has three main LLM surfaces:
An interactive chat agent (trip planning, "what am I looking at", edits to itineraries) — a server-side harness with ~30 tools, serving iOS, Android, and web. A content pipeline that writes and translates guides for 170,000+ pages across 11 languages — factual descriptions of monuments, cities, practical visitor info. Background jobs — classification, event discovery, daily briefs, and the judges for the evals themselves. Nothing here is "IQ 180" work, to borrow the taxonomy from French-Owen's post. It's almost all what he calls token-spewer work: high-volume, iterative, verifiable. Which is exactly the lane where small models were supposed to arrive. So did they?
Rule 1: route by workload, not by leaderboard The single most useful policy change I made was to stop asking "which model is best" and start asking "which model passes this workload's gate." My evidence hierarchy, in order:
My own task-specific evals and production telemetry. The exact output contract and the cost of failure at that call site. Current provider latency, availability, and tool-call reliability. Public leaderboards — last, and only to pick candidates for a bakeoff. Today the fleet looks roughly like this: fast Gemini Flash variants handle classification and interactive reads (reasoning off or low), GPT-5.6 Luna writes the factual content, a 120B open-weights model does one-shot intent routing and reply checking, DeepSeek V4 Pro is allowed exactly one tool-grounded repair per conversation, and a frontier model (GPT-5.6 Sol) sits at the end of the ladder as a bounded cross-family recovery — one attempt, specific trigger conditions, never the default. Claude judges the evals.
Five model families, each pinned to a role, each promoted only through a gate. No auto-router. Routers optimize for benchmarks; my gates optimize for my failure modes.
Rule 2: the bakeoff, and what the judges actually measured The eval that changed my pipeline was a bakeoff for the factual-writing stage — the model that turns a researched fact packet into the guide text a narrator reads aloud. The contract for that stage is strict: write from the packet, because every sentence must be traceable to a source.
Method: each candidate duels the incumbent, blinded, on the same research packets. A/B slots alternate so position never correlates with identity. Two independent judges (Claude Sonnet 5 and Claude Opus 5) score factuality and style, and count claims unsupported by the packet. Hold that phrase — it matters.
Two rounds, and I'm labeling them precisely this time because my first write-up blurred them:
Exploratory round, n=5 packets: the incumbent (DeepSeek V4 Pro) against six challengers including Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, and GPT-5.6 Luna/Terra/Sol. Confirmation round, n=20 fresh packets across 20 countries: Luna and Terra vs the incumbent only. Confirmation-round results (the load-bearing ones):
Model (n=20 round) Factuality (Opus judge) Head-to-head Schema p50 latency $/packet GPT-5.6 Luna 4.18 17/17 wins 19/20 42.6s $0.0058 GPT-5.6 Terra 3.83 18/18 wins 20/20 32.1s $0.0353 DeepSeek V4 Pro (incumbent) 2.09 0/35 18/20 144.7s $0.0310 Luna won every duel under both judges, 5.4× cheaper and 3.4× faster than the incumbent. That result stands. What needed a second look was the interpretation of the "unsupported claims" counts — especially the exploratory-round observation that the expensive Anthropic models racked up the highest counts (Opus 5 and Sonnet 5 around 13 per judged output, against Luna's ~3.6).
Rule 3: I fact-checked my own eval. Most "hallucinations" were true. "Unsupported by the packet" is not the same thing as "false." A model with strong world knowledge can write things the packet never mentioned that are perfectly correct. So I ran a verification study on the exploratory round: a fixed extractor listed every packet-unsupported atomic claim in all 35 outputs (7 models × 5 subjects) — 406 claims — and a web-search-grounded verifier fact-checked each one as TRUE, FALSE, or UNVERIFIABLE, with sources. Total cost: $3.39.
Model (n=5 round) Unsupported claims Verified TRUE Verified FALSE % true False claims / 1k words GPT-5.6 Terra 31 29 1 94% 0.30 GPT-5.6 Luna 22 20 2 91% 0.54 GPT-5.6 Sol 53 49 4 92% 1.28 Claude Opus 5 107 95 9 89% 1.76 Claude Sonnet 5 70 57 10 81% 1.95 DeepSeek V4 Pro (incumbent) 95 79 13 83% 3.24 Claude Haiku 4.5 28 14 10 50% 4.68 Three corrections to my original story, in increasing order of importance:
First: 81–94% of what my judges called fabrication was true. Opus 5's 107 ungrounded claims include "the Neuer Markt was historically called the Mehlmarkt" (Vienna city records), "Georg Raphael Donner died in 1741" (Britannica), and "the Tiled Kiosk was built in 1472 by Mehmed II" (Archnet). All true. The judges flagged one output for "inventing a Black Head Lighthouse (1936)" — the lighthouse the article was about, whose light was established on 21 February 1936 per the Commissioners of Irish Lights. The packet just didn't mention it.
Second: the raw counts were partly a verbosity artifact. Anthropic models wrote ~1,020–1,070 words per output; Luna wrote 746, Sol 627. Count claims instead of rates and the verbose models look worse. Normalized per 1,000 words, the ungrounded rates for Opus (12.8), Sonnet (12.7), and my own incumbent DeepSeek (12.9) are a three-way tie. The claim "frontier models ignore the grounding contract most" doesn't survive normalization; "Luna and Terra follow it 2.5× better than everyone" (4.7 and 5.3 per 1k words) does.
Third — and this is the part that saved the conclusion: the ranking on actually false claims still puts the small OpenAI models first. Terra emits one false claim per 3,300 words; Luna one per 1,900. My incumbent emitted one per ~300 words — six times Luna's error rate — and it wasn't padding its answers with bonus knowledge; its extra claims were wrong more often (17% vs Luna's 9%). And the biggest surprise: the real fabricator in the field was the cheapest Anthropic model. Half of Haiku 4.5's ungrounded claims were false — it told me Donner was 64 when he designed his fountain (he died at 47), that the statues were "replaced with clothed versions" (they weren't), and that "scholars remain uncertain" about a replacement date that's firmly documented as 1873. Fabrication risk doesn't track price in either direction.
A small gallery of real failures Verified-false claims, so you can calibrate what "wrong" looks like in this domain:
DeepSeek V4 Pro (the incumbent I retired): claimed Osman Hamdi Bey brought the Alexander Sarcophagus into the Tiled Kiosk (it famously didn't fit — he built a new museum for it); dated the Süleymaniye Mosque's opening to August 1557 (it was 15 October); asserted Mellifont's monks "ate no meat" (the Rule permits it for the sick). Claude Opus 5: gave Donner a pauper's burial he didn't have; put fish in the hands of fountain figures that hold an oar and a trident; attributed the Tiled Kiosk's tiles to the wrong technique (cuerda seca vs banna'i mosaic). Claude Haiku 4.5: see above — 10 of its 28 ungrounded claims failed verification, several unambiguously invented. GPT-5.6 Luna: two false claims in 3,700 words, and the more interesting one wasn't invention but arithmetic — "the Carboniferous ended more than 100 million years before the first dinosaurs" (it's ~55–65M). Sol made a near-identical temporal-reasoning slip. Small models' residual errors in my data skew toward reasoning-under-composition, not fabrication. One caveat cuts the other way: verifier granularity. A few FALSE verdicts are pedantic — Sonnet locating Marmara Island "in the Aegean" (it's the Sea of Marmara), Opus calling the Tiled Kiosk "Istanbul's oldest secular building" (oldest Ottoman secular building). I left them in the counts because narration read aloud as fact should survive pedantry, but the per-model gaps are larger than the nitpick share.
What the metric actually measures, and why I keep it If most ungrounded claims are true, why enforce packet-grounding at all? Because Audiala's product promise is traceable narration — every claim in a guide should be checkable against collected sources, and a 9–17% error rate on free-floating knowledge is far above what I'll read into someone's ears at a monument. The eval's mistake wasn't the metric; it was my language. "Unsupported" measures contract compliance, and the contract exists because verification, not vibes, is what lets a one-person shop publish 170k pages.
The study also exposed a real weakness on my side of the contract: one packet (the Tiled Kiosk) was actually about the parent museum complex, and barely covered the kiosk itself. The models that knew the building's real history got penalized for being right about the subject. When the packet is thin, the fix is deeper research, not a model that knows more and free-associates — but the eval can't tell those apart unless someone checks, which is exactly what this post did.
Rule 4: the harness is the moat A commenter on the HN thread (swatcoder) predicted a "cambrian explosion of restrained but efficient model+harness-tuples … rigged to deliver on narrowly scoped but idiosyncratically-shaped tasks." That is, verbatim, what my agent turned out to be. The chat agent runs Luna-class and Flash-class models and passes 94% of a 53-scenario eval suite (13/13 on the safety-critical ones) at about $0.0007 per scenario — because the harness, not the model, does the heavy lifting:
Verify-after-write. The model never gets to claim "done." The harness re-reads the store and confirms the mutation actually happened. A finite repair ladder. First answer, plus at most one eligible repair by a designated specialist. No unbounded retry loops. Deterministic replay as a CI gate. Recorded scenarios re-run for $0 on every pull request. Live model runs only on tagged releases. Promotion gates. A new model ships only if it beats the incumbent on the exact workload with no regression on safety cases — against a frozen pre-change control. And I'd now add a fifth, earned this week: audit your own judges. My eval logged every packet hash, every output, every judge verdict — which is the only reason I could re-open it, re-derive the claims, and fact-check them for $3.39. An eval you can't re-litigate is an opinion with a spreadsheet.
Limitations, before you ask The frontier-model comparison is n=5 packets (35 outputs, 406 verified claims). The Luna-vs-incumbent result is n=20 and judge-consistent; the Opus/Sonnet/Haiku numbers are directional. My two judges agreed on the factual winner only 73% of the time at n=5 (87% at n=20). I only act on clean sweeps that hold under both judges. The judges were Claude models scoring a field that included Claude models — and penalized their own family, so whatever bias exists didn't favor the home team. One model extracted the claims and one web-grounded model verified them; both Gemini, neither a contestant. Spot-checks by hand (including the lighthouse) matched. Style scores and factuality moved in opposite directions in every round: the prettiest prose carried the most ungrounded content. Judge both axes or drift toward eloquent fiction. What still needs the frontier I'm not a small-model absolutist. Frontier models still earn their price as the bounded last-resort recovery, as eval judges (with the reliability caveats above), and — the biggest line — as the coding agents that build the system itself. And the corrected data adds nuance to French-Owen's taxonomy: the frontier models weren't hallucinating, they were overflowing — bringing true knowledge the contract didn't ask for. For discovery work, that's the point. For token-spewer work under a verification contract, it's a liability you pay 20× more for.
So the conclusion survives its own peer review, sharpened: for grounded production writing, the small models are simultaneously the cheapest, the fastest, the most contract-compliant, and the least likely to state a falsehood — my incumbent had 6× Luna's verified error rate, and the worst fabricator in the field was small too. Small models have arrived, but don't take your eval's word for what "hallucination" means. Check.


---

## About Audiala

Audiala provides AI-powered audio guides for 1,100+ cities. Free first 5 guides, works offline, 11 languages.

- App (iOS): https://apps.apple.com/us/app/id6446038181
- App (Android): https://play.google.com/store/apps/details?id=com.audiala.audioguide
- Download router: https://audiala.com/en/download/