Jev in Production: What Worked, What We Rolled Back, and What We Measured

On September 17, I tested Jev on 28 labeled routing examples from Audiala, the travel app I build. With a policy-aware rubric, it classified 27 correctly. Its uncertainty on the remaining example gave us a sensible clarification path. The model call cost about a quarter of our incumbent's.

Later that day, we switched the production router to Jev. Twenty-three minutes later, we rolled it back.

The intent labels were not the main problem. Our router also supplied the city, conversational referent, and flags that determined which tools the assistant could use. We had tested the headline output and underestimated the rest of the contract.

That first week produced useful wins too: better rankings on a small recommendation benchmark and a cached place-eligibility layer that replaced a brittle preparation dependency. But the useful story is the difference between a promising decision model and a successful integration.

Where a decision model fits

TypeSafe's Jev takes state plus typed questions. Its primitives are Choice for selecting an option, Score for an ordered rubric, and Noul for a proposition scored from zero to one. It returns structured decisions rather than generated prose.

That is a natural fit for parts of Audiala. We already have a catalog of real places. Often the question is which of those places suits a traveler, whether two records describe the same subject, or which available action matches a request.

Restricting the answer to supplied candidates removes one way to invent a place. It does not make the selection correct. The catalog can be wrong, the relevant candidate can be missing, and a model can confidently choose the wrong entry.

Our working division is straightforward: models judge relevance and write language; code resolves identities, checks constraints, authorizes changes, and verifies the result. Jev gives us another way to implement the judgment part.

The measurements below are our September 17–22 experiments, principally on pinned jev-1.13.0. They are small workload studies, not a general model leaderboard or a claim about today's pricing.

The first probe: confidence cannot know an unwritten policy

The routing corpus contained 28 turns across 11 languages. With our existing rubric, Jev and the incumbent, gpt-oss-120b through OpenRouter/Cerebras, both got 26 right. They missed the same two dining cases.

Those cases encoded product decisions: an ambiguous sightseeing-and-food request should trigger clarification; an explicit food-only crawl belongs on a read-only path. The production system handled those boundaries with deterministic rules. Our model rubric did not explain them adequately.

Jev was confidently wrong: 0.99 and 0.97 confidence on those two cases. After we wrote the policy into the option descriptions, one became correct and the other became uncertain, with confidence 0.37. A 0.5 threshold sent it to clarification.

That gives 27/28 classification accuracy, with an appropriate abstention on the last example. Calling it “100% accurate” would hide the mechanism that made it useful. The rubric and threshold were also adjusted on this corpus, so the result needed fresh validation.

September 17 routing probe	Correct labels	Median call latency	Mean cost per call
Incumbent, production prompt	26/28	334 ms	$0.000113
Jev, original rubric, cold connection	26/28	652 ms	$0.000026
Jev, revised rubric, warm connection	27/28	250 ms	$0.000027

The last two rows change both rubric and connection reuse. They are not a clean experiment on either variable alone. They did show why we needed persistent connections: a remote call from Switzerland spent a material fraction of its time getting to the service.

For developers, the lesson is to put product policy into the decision definition and evaluate abstentions separately from correct answers. A confidence threshold is a parameter to validate on your traffic, not a universal safety setting.

Why the router rollback was the right outcome

There were actually two integration mistakes before we reached a meaningful comparison.

First, we wired the older router path. Over the preceding seven days, it had handled three turns; the newer engine had handled 1,624. Flipping a configuration value did not prove that the serving path used it. Routing telemetry exposed the mistake.

Second, the router's real contract was much bigger than intent. A generative router could return a city name; Jev needed a menu of resolved city candidates. Flags such as dining and app control affected the available tools. A referent such as “this plan” determined the target. Matching the top-level label was insufficient.

Once the active path was wired, the first production switch regressed the repeated end-to-end suite, including three newly stable failures. We reverted it and added comparisons for the complete router output.

Even after those repairs, the latency argument did not hold:

Later full-suite comparison, 65 scenarios × 3 runs	Incumbent router	Jev router
Scenarios passing all three runs	38	33
Passing scenario runs	121/195	117/195
Router median	382 ms	294 ms
Full-turn median	8,677 ms	9,262 ms

These are results on a difficult regression suite, not organic-user success rates. They show why the component benchmark was not enough: we saved 88 ms in a roughly nine-second interaction, while the full turn got slower in this run. In the previous run, the end-to-end difference had gone the other way.

We also tested a hybrid: Jev first, then the incumbent when confidence was too low. The router remained faster, but the end-to-end improvement again failed to repeat. A slow Jev request could consume its timeout budget before fallback even began.

We kept the incumbent router. The replacement capability remains available for another measured comparison; the early probe did not earn a production default.

Ranking was a better fit—and still needed an honest comparison

Our recommendation test used four cities, one traveler persona per city, and 44–48 catalog candidates per pool. A blinded LLM judge assigned relevance labels. We compared the production Voyage reranker with Jev scoring each candidate and with a single-Choice variant.

The metric was nDCG@8: how closely the first eight recommendations matched the judge's ideal relevance ordering. Higher is better; one is ideal under those labels.

City and persona	Voyage	Jev per-candidate Score
Karlovy Vary, architecture	0.865	0.927
La Rochelle, young family	0.876	1.000
Sion, outdoors	0.569	0.843
Riquewihr, food and wine	0.822	0.802
Mean	0.783	0.893

The Riquewihr result matters. Most of that pool consisted of half-timbered houses. A ranker cannot supply wine experiences that retrieval never found. Jev was slightly worse there.

Per-candidate scoring was more consistent than our single-Choice variant, which averaged 0.795 and performed poorly for the family in La Rochelle. For a ranked slate, an explicit rubric for each candidate gave us more useful evidence than asking for one choice.

This was a quality tradeoff, not a speed win. Jev's scoring calls took 536–888 ms; Voyage took about 305 ms. Jev cost roughly $0.00037 per rerank. The path could accommodate that latency, and we enabled it with Voyage retained as fallback. One of the first six production calls received a TypeSafe overload response and used that fallback.

There are two important limits to the comparison. Four cities are too few to establish broad superiority. And Jev and the judge saw structured persona information while Voyage received our existing production query string. Improving that query might narrow the gap. We measured replacement of our implementation, not each vendor's best possible configuration.

Good stops do not automatically make a good itinerary

A separate probe used four cities and two traveler scenarios per city. Jev judged the candidate stops; a small code-based assembler grouped and ordered them. Gemini 3.8 Flash produced a one-shot plan from the same candidate pools.

Eight city–scenario cases, LLM-judged	Jev + geographic assembly	Gemini one-shot
Mean relevance label of selected stops, 0–3	2.50	2.46
Judge-labeled must-sees selected	28/45	27/45
Overall plan score, out of 50	39.1	42.2
Model cost per plan	$0.0015	$0.0043

The stop selection looked competitive. The complete plans scored lower. With eight cases and an LLM judge, the tiny differences in stop relevance are not evidence of a decisive win.

The most actionable result came from changing only our assembly code. Adding better geographic grouping raised the Jev-based plan score from 36.6 to 39.1 without changing its model answers. Our first assembly had happily paired attractive architecture stops far apart.

That experiment justified investigating a candidate judge. It did not justify replacing the planner. Later production work also taught us to rank the candidates the assistant reads, rather than silently replace the stops it chose after reading the traveler's constraints.

Cache decisions where the evidence is stable

Another useful integration was place eligibility: is this catalog entry suitable for a visitor recommendation, what kind of place is it, and how important is it?

Our previous prepared city index could leave a draft with no usable candidates when a city or locale artifact was absent. We replaced that dependency with decisions computed on demand and cached by canonical place identity, with a digest of the evidence and decision configuration. Editorial overrides take precedence.

In one Cork measurement, the first call took 1.8 seconds and a subsequent call took 63 ms. That is a cache result, not evidence that Jev itself answers in 63 ms.

The cache added its own operational problem. A rubric change invalidated decisions; exhausted provider credits prevented recomputation. Discarding every old decision would have abruptly removed filtering. We added a serve-stale path, exposed stale coverage, and documented prewarming before a rubric change.

The failure policy here is specific to a recommendation filter. An unanswered eligibility question need not remove a real place from the pool. That is very different from permission to edit someone's saved data, where a missing authorization must still block the write.

The integration checklist I would start with next time

1. Trace the serving path. Record the engine and fallback reason on real turns. A configured experiment that silently falls back is measuring the wrong system.
2. List every consumed output. Include identities, flags, references, and UI choices—not just the intent label.
3. Separate the gates. Test the decision in isolation, then the whole workflow, then the client outcome. Keep a fresh holdout after rubric tuning.
4. Measure the time budget. A consistently faster 300 ms component can still be the wrong optimization inside a ten-second turn.
5. Version decisions. Bind cached results and thresholds to the model, rubric, and input evidence. Make missing and stale coverage observable.
6. Give fallback a deadline. Measure the combined primary-plus-fallback latency, and preserve deterministic authorization regardless of model output.

As of the September 22 evidence cutoff, Jev had earned specific production roles in recommendation reranking and cached place decisions. The main intent router remained on its incumbent. History elision was in shadow mode; the additional tool-risk classifier was off. Neither experiment had earned a claim of measured user benefit.

The improvement I want next is a better completed journey for the traveler. A lower model invoice or a perfect small corpus is useful only when it gets us there.

Companion article: What our agent harness learned from green tests and broken journeys. Our earlier small-models study explains the workload-specific evaluation approach behind these experiments.

Measurement note: these are internal Audiala probes and dated deployment observations. Costs are recorded model-call costs, not total infrastructure or engineering costs. Ranking and planning labels came from an LLM judge; router labels came from our product-policy corpus. Raw internal traces are not a public benchmark release.