Does the Selected Object Reach the Reader? Auditing Identity Handoffs in Grounded Language-Model Pipelines

Picture a person asking a question about a table on a web page. A first step reads the table and picks one linked page. A second step turns that pick into a search and returns the five pages it ranks highest. A third step reads those five pages and writes the answer.

The model is handed the question, the table the question came from, and those five passages. It never sees the picked page unless the second step put it in the list. So a pipeline can fail in three different places: the pick was wrong, the pick was right but got lost on the way, or the pick arrived and was ignored. One score for “did the right page show up” cannot tell those apart, and they need three different repairs.

Benchmarks come with an answer key that links one page per question. A system picks its own. They are often the same page, and sometimes they are not, and then the score depends entirely on which one you asked about. Switch the search rule below and watch the two swap places.

Across 1,792 records where both pages exist in the corpus, the two disagree on 329, drawn from 160 of the 600 questions. Searching with the plain question, the two give different verdicts on 106 records, or 5.9 percent. Searching with words taken from the picked page’s own title, they differ on 213. Which number you report depends on which page you scored and what you typed into the search box, so a report needs to say both. In the checks the paper runs, swapping the target never reorders the five search stacks, for any model family or cutoff. What changes is the number you are entitled to report.

Take only the 1,463 records where both pages agree, so there is nothing to argue about. Hold that page fixed. Give the seven search rules the same words to search with. Then count how often the page is missing from the five that reach the model.

On these records, looking the page up by its key never loses it, and neither does matching its exact title. The search rules lose between 14 and 389 of the same 1,463. Of the 14 that the best of those seven loses, six sit just below the line and a wider window would catch them; the other eight are not in the top twenty at all.

Recorded runs that differ in whether the shared page was among the five tell a consistent story.

64 questions were chosen before any new run happened: cases where the model had been right with the shared page and wrong without it, in both of the recorded comparisons. Each was asked again three times with nothing removed, and three times with the picked page removed. 59 of the 64 also got a third pass with a different page removed instead.

The comparison in the middle is the one that matters. On the 59 questions that have both, cutting a passage of similar length that does not hold the answer leaves the score where it was, and the interval on that difference includes zero. Cutting the picked page collapses it. The table the question came from is still in the prompt either way.

The released instrument is a file rather than a table. It names the page being checked, the field the returned IDs come from, the cutoff, the rule for what counts as returned, and the full list of records the run is supposed to contain. A reference program reads that file alongside the run and gives one verdict per record and one status for the whole run.

Each of those verdicts points somewhere different: a page found but outside the cutoff is the retrieval step, a missing or malformed list is the logging, and a duplicated or unplanned record is how the run was assembled. A single percentage merges them. The released package carries one such record per result, each checked against the frozen rankings and answer records it was built from, and the check runs with no network connection and no API key.

One corpus: English Wikipedia tables and the pages they link. One recorded output per model and question, so nothing here says how much a picker’s choice wobbles if you run it again. Counts pooled across the three model families are not three independent samples of the same questions.

The 329 records where the two pages disagree carry no judgement about which page would have supported a correct answer. Nobody read them. And the deletion result holds for the 64 questions it was run on, which were chosen precisely because they were clear-cut.

The released checker confirms that the bytes it was given match their hashes and that the records it saw match the list it was promised. It does not establish who produced a log or when. That needs controls outside any evaluator.