Green Tests, Broken Journeys: Lessons from Building Our Agent Harness

We made our travel assistant display its answer sooner. A live probe showed the first text arriving 0.5–2.7 seconds earlier across five turns. The server checks passed.

Then the Android journey test tried to answer a clarification question and could not find a matching button.

The assistant was asking which kind of tour the traveler wanted. The buttons said “View itinerary” and “Add more stops.” We had improved latency by breaking the conversation. We reverted the change.

I build Audiala, an audio travel guide with a conversational assistant across iOS, Android, and web. By “harness,” I mean the runtime around the model: routing, context, tools, authorization, persistence, streaming, recovery, and the evidence used to decide whether a task is complete. This article is about that product runtime, rather than the coding agents we use to develop it.

Our earlier small-models article described the value of this machinery. September supplied some less flattering—and more useful—examples of where it failed.

A button can be part of the meaning

The latency change looked reasonable. Our model sometimes spent its entire first round calling a follow-up suggestion tool, then wrote the actual answer in round two. In three of ten traced turns, the buttons arrived roughly a second before any prose. Another two turns spent an extra suggestions-only round after other tool work.

We tried withholding the suggestions tool in the first round. The answer appeared earlier. But a clarification often ends after that first round, so the model never got a chance to supply the choices paired with its question. Server-generated fallback buttons answered a different question.

The phone test caught it because it did what a traveler would do: look for the offered answer and tap it. A server trace containing valid text and valid buttons was not enough.

We had treated suggestions as decoration. In that conversation, they were executable answers.

The general design rule we took away is to keep a clarification's question, labels, and submitted values together. A choice needs to retain its meaning from model output through rendering to the next request. Moving its generation elsewhere requires preserving that relationship, not merely producing plausible buttons.

A runtime migration can silently delete product policy

Our newer engine originally ran beside the older one. It reused the domain tools and offered a thinner streaming loop. That was a sensible migration boundary, but some deterministic behavior lived outside the tools.

One example was a pre-loop rule for ambiguous dining-and-sightseeing requests. Before retrieval or writing, the old path asked a concrete scope question. The new engine returned before reaching that block of code.

The regression check stayed green because it only booted the old path.

On the new path, whether the assistant asked or guessed became model-dependent. It could produce monument cards and a restaurant recommendation without ever asking the question. Restoring the deterministic gate gave us a structured clarification in measured live probes of 147–641 ms, with no model call.

The fix included running the behavioral check against both engines. We then looked for the same migration gap in nearby reference and location recovery rules.

For a similar migration, I would write a small parity table before moving traffic:

Behavior	What the check must observe
Ambiguous request	Structured question before any mutation
Missing target	No guessed record identity
Approved action	Same exact target after resume
Cancelled or expired action	No delayed mutation
Client navigation	Actual supported destination and execution result

The table describes externally observable contracts. A shared helper does not prove that both entry points call it.

A verified write can still leave the request unfinished

Another failure started with a request to build a regional tour in French-speaking Switzerland.

The assistant found cities and added three destinations to a Journey. Then drafting the first destination's local Plan hit a guard with no suitable approval-resume executor. The turn ran out of rounds. The final receipt accurately said that three destinations had been updated, but omitted the blocked drafting step.

Nothing in that sentence was fabricated. The traveler still did not have the requested tours.

This exposed an important limit of verify-after-write. You must check the requested outcome as well as each successful mutation. A receipt for creating a container does not prove that its contents exist.

The investigation found several interacting problems: dates pushed a destination draft into the wrong confirmation path; switching the active workspace made the next Journey operation lose its target; a one-day draft could fail a multi-day stay check; and the reply path omitted blocked calls when summarizing successful evidence.

We made Journey and destination identities explicit, made preparation capable of drafting multiple destinations in one bounded operation, and surfaced blocked work alongside successful receipts. A later founder retry found another gap: the city guard understood a Trip's city but not a Journey's destination cities. That needed its own fix.

The resulting two-turn live scenario proposed verified city options, accepted a selection, and created three local Plans. That was evidence for the repaired scenario, not proof that every regional request now works. Founder acceptance remained a separate gate in the September 22 record.

For outcome reporting, I now ask three questions: what did the user request, what durable objects exist, and what required step is still missing? The reply must preserve all three answers.

Our eval could not perform the operation it needed to test

The regional-tour investigation also explained a hole in our coverage. The eval suite supplied a user ID in development mode, but no traveler access token. Journey operations ran under the traveler's own database authorization. The suite could not exercise those operations at all.

It contained zero Journey scenarios.

We changed the runner to sign in each evaluation user and send a real access token, then inspected the result through the same traveler-scoped snapshot operation used by the feature. The first authenticated runs immediately found more defects, including missing tool-risk registration and incomplete multi-destination preparation.

That does not mean every unit test needs a live login. It means at least one integration path must use the same identity and permission boundary as the feature it claims to cover. A privileged fixture can prove the storage shape while bypassing precisely the authorization behavior that fails in production.

Record why a tool failed

A September review of 224 real app turns found 51 that ended as an answer after a failed tool call. There were 24 failed itinerary-drafting calls in that slice.

Those counts identify a problem worth investigating; they do not establish 51 false completion claims. Some answers after failures can be honest recoveries. Unfortunately, our ledger often recorded only the tool name and ok: false. It discarded the reason even when the loop had it.

We added a bounded failure reason and preserved which guard blocked the operation. That made future failures diagnosable. It could not retrospectively explain the missing details in the earlier 24 drafts.

A useful turn record needs enough information to distinguish an ambiguous request, an authorization refusal, a provider failure, an empty candidate pool, and a write whose result did not verify. It also needs the serving revision and traffic source. Our evaluation traffic can dominate the ledger; mixing it with real app turns produces misleading product metrics.

Keep the record narrow and scrubbed. The objective is a causal explanation, not a second unbounded store of user conversations.

Optimize work the system does not need to repeat

Our Jev experiments repeatedly made a router call faster without delivering a repeatable improvement to the full interaction. In one plan/edit evaluation window, the median router took 338 ms inside a ten-second turn.

The more promising target was the surrounding work. A later seven-day production-ledger query found 656 identical repeated search_monuments calls among 5,263 calls to that tool: 12.5%. That ledger slice included evaluation activity, so it is not a rate measured solely on organic travelers.

We introduced in-turn memoization for eligible reads. Its boundaries matter more than the cache implementation:

• The key includes the tool name and canonicalized arguments.
• A non-read operation invalidates the memo.
• Errors are not cached.
• Presenters, confirmation previews, and control pauses are excluded.
• Reusing a result preserves the evidence and client event behavior.

“Read-only” does not necessarily mean “safe to skip.” A tool can leave the database untouched while presenting a card the client needs.

The dedupe switch was enabled on September 22. The repeated-call count identifies an opportunity; it is not a measured post-release latency saving. Writes and confirmation retries remain outside this optimization.

Shadow mode has to preserve the control

We also added experimental history elision: score older tool observations for relevance, then potentially replace low-value results with short stubs. The design protects the traveler card, recent exchanges, pending confirmations, and receipt-bearing results.

We started in shadow mode. It records what it would remove while leaving the model's actual history unchanged. A follow-up moved shadow scoring off the turn's critical path, because an experiment that changes no content can still slow the user down.

At the evidence cutoff, early production events showed the switch was read and fresh sessions were under budget. That does not establish useful compression, lower cost, or preserved quality on long conversations. Those need their own replay and production evidence.

This is a useful distinction whenever you add a judge or classifier: installation, activation, and demonstrated benefit are three different milestones.

Test the boundary you are claiming

We now use several kinds of evidence because each catches a different failure:

Evidence	What it can establish	What it leaves open
Unit tests and deterministic replay	Rules, state transitions, known regressions	Live model and provider behavior
Repeated live scenarios	Variance, tool use, end-to-end persistence	Actual phone or browser interaction
Authenticated client journey	Rendered choices, taps, navigation, reopened saved state	Broad user success across real requests
Organic traffic review	Failures and demand in actual use	Causality without controlled comparison

No single percentage summarizes all four. A change can pass the first two and still fail because a traveler cannot answer its question.

For a small team, the most valuable harness work has been quite concrete: retain the exact target through a resume, keep a button bound to its question, read back the saved object, show the unfinished part of a request, and record the reason a tool stopped. Those are the details I want to see before trusting the next green test run.

Evidence note: these incidents and measurements come from Audiala's internal September 2026 reviews, live probes, client gates, and release records, with a September 22 cutoff. The regional-tour request is a paraphrase of a founder test; no customer transcript is reproduced. Newly enabled deduplication and shadow elision are described as experiments, not established production gains.