The Right Answer About the Wrong Place

The place in our catalog was Kraftwerk Peenemünde, a power station in Germany. The English audio-guide text stored against it said:

Standing here, you're looking at the Blockhaus d'Éperlecques. The Watten bunker. Codename: Kraftwerk Nord West.

That is a different site, in northern France. The French-language guide made the same substitution.

We found it during a September 10 audit of generated narrations for Audiala, the audio travel guide I build. The wrong building was real. The retrieved article was real. Its connection to the requested place was plausible enough to survive our pipeline.

This was an entity-resolution failure upstream of writing. It exposed a question our grounding checks were not answering: does the source describe the subject we were asked about?

The fallback that changed the subject

Our pipeline started with a canonical catalog identifier: Q14906977, the Wikidata item for Kraftwerk Peenemünde. It used that item's Wikipedia links to gather source material, condensed the material into a fact sheet, and passed the sheet to a narrator.

The archived source inventory for this incident contained a German Wikipedia article and a Commons link, but no English Wikipedia link. There was relevant German material available. The fallback ran because English was missing, not because all sources were missing.

It searched English Wikipedia for the place name and city, requested one result, then merged that result and its translated article links into the source set. It did not verify that the returned article belonged to the requested Wikidata item.

Our incident record identifies the returned hit as Blockhaus d'Éperlecques, associated with the wartime codename Kraftwerk Nord West. The vocabulary and historical connection made it a plausible search result. The French site's own history discusses its connection to Peenemünde; a connection between two places does not make them the same place.

The failure crossed these boundaries:

The English and French outputs repeating the same error were not independent confirmation. Both descended from the same contaminated source set.

A better writer could have noticed the contradiction. But requiring a downstream model to detect a broken identity join was a poor substitute for checking the join itself.

Why grounding checks missed it

Our narrator bake-off judged outputs against the pipeline's own fact sheets. That answers a useful question: did the writer follow its supplied material?

It does not independently check whether the supplied material belongs to the requested place. A wrong source can produce a wrong fact sheet, followed by an obedient narration. Comparing the last two artifacts can reward consistency while missing the original error.

The later accuracy audit used separately assembled subject sources, including the requested Wikidata item and its linked articles. That gave it a reference independent of the generator's fact sheet. It was still an audit with fallible sources and a model judge; it was asking a different question.

The same distinction matters in a cited chatbot. A valid citation can establish that a document contains a statement. It cannot, by itself, establish that the document is about the user's intended entity. Our product delivers audio; we are not claiming that listeners saw citations and trusted them. The weakness was in our internal grounding and evaluation, and it transfers to systems that do display sources.

These are three separate checks:

• Subject: does the evidence describe the requested entity, or explicitly describe a relevant relationship to it?
• Support: does the evidence support the claim?
• Fidelity: did the output preserve the supported claim accurately?

Passing the last check cannot compensate for failing the first.

The patch—and what it actually guarantees

The September 10 patch passed the expected catalog ID into the name-search fallback. When fetching the candidate page's language links, it also requested its Wikidata association through MediaWiki page properties:

action=query
titles=<candidate article title>
prop=langlinks|pageprops
ppprop=wikibase_item

MediaWiki documents the page-properties API here. The important architectural point is where the returned identity comes from: the candidate page's metadata, separately from the catalog's requested ID. Copying the requested ID onto a retrieved document and comparing it back would prove nothing.

The guard's behavior can be reduced to this:

if looks_like_qid(expected_id):
    if source_id and source_id != expected_id:
        return {}  # reject the candidate and its language links

The incident record reports a same-day check in which the French article was rejected while the existing German article remained available. That was a single live check, not a result from the replay below. We could narrate in English from material about the correct subject in German. A missing output-language article did not justify importing a different subject.

But this is a known-mismatch guard, not a complete identity gate. If the candidate supplies no ID, the inner condition does not fire. If the expected ID is absent or does not have the recognized shape, the old permissive path remains.

The historical function exposed a limit that our wiki summary had obscured: missing identity metadata still allowed admission.

A small replay that exposes the boundary

For this article, we extracted the name-search function from immediately before and after the patch and supplied fabricated MediaWiki responses. The function itself was unchanged; network responses were controlled. Six cases across the two revisions produced these results:

This is characterization of a specific historical code path. It is not a fresh live Wikipedia test, an estimate of production accuracy, or proof that every downstream caller handles rejection correctly. The fake mismatched ID is simply a fixture value; it is not presented as the French bunker's actual identifier. The separate sufficiency change below was checked against its patch, not replayed with these fixtures.

The missing-ID cases are the useful part of this table. A test suite containing only “same” and “different” would validate the patch while leaving the permissive behavior invisible.

A stricter interface would return match, mismatch, or unresolved. Only match would admit an article as the primary subject anchor. Unresolved could trigger another source lookup or review; it should not silently become a match. A page about a related entity can still be useful, but it needs an explicit role as contextual evidence rather than being substituted for the subject.

The downloadable Python example models the historical guard and this stricter policy, with a runnable decision matrix. It uses the standard library:

python3 identity_gate_demo.py

It is an educational example, not our production module or a complete resolver. It also tests a deliberate limitation: two identical invented Q-shaped identifiers compare equal. Syntax validation does not establish that an entity exists or that its catalog assignment is correct. Redirects and merged identifiers need canonicalization; identifiers from different namespaces need a verified mapping.

More text was not more evidence

A second path could bypass the need for a substantial article altogether. Our old sufficiency condition was:

enough = wiki_chars > 300 or len(search_snippets.strip()) > 200

That let a sufficiently long name-search response authorize a narration. For some memorial records, snippets described unrelated people with the same name. There was no substantial article or reference text anchoring the biography to the person commemorated.

We changed the condition to require article text or fetched reference text:

enough = wiki_chars > 300 or len(reference_text.strip()) > 600

Removing the snippets-only route narrowed the problem. It did not make character counts an identity check, and a long article admitted through a permissive identity path remains a problem. Source admission and source sufficiency have to work together.

There was a real coverage cost. The same change also suppressed a guide that had happened to be correct despite its thin grounding. We accepted that loss for the sensitive memorial cases, and cleared affected published text and audio together. A repaired text field would not stop an already-generated recording from speaking the wrong biography.

That manual cleanup is separate evidence from the code fix. Changing future retrieval does not repair previously generated outputs. Nor does a generator returning no result prove that a caller's fallback will safely withhold narration; that behavior needs its own check.

What I would put in the next eval

The smallest useful addition is a wrong-but-plausible source. Give the system the correct target identity and a highly relevant-looking document about a different entity. Check that the document cannot become the subject anchor, even if a writer could faithfully summarize every sentence in it.

Then add the cases that make a policy uncomfortable: missing identity metadata; a correct source in another language; a parent site instead of the requested building; two namesakes; and a previously generated artifact that must be withdrawn after its source is rejected.

For each retrieved document, preserve the requested entity, independently resolved source entity, source URL or revision, and admission decision. Keep those attached when creating chunks and fact sheets. That makes a wrong join inspectable before fluent prose hides it.

This changes the evaluation question from “did the answer use the documents?” to “were these documents entitled to speak for this subject?” We had measured the former while assuming the latter. The guide about the French bunker was the cost of that assumption.

Evidence note: this is a September 10, 2026 incident retrospective, checked against archived narration/source records and the historical patches. The six-case function replay was run for this article on September 22. The three-way policy is a recommendation, not a claim that it is deployed. The cover is a fictional illustration, not a depiction of either historical site.