0
Your RAG copilot can't count — stop letting it try
TL;DR: A RAG copilot should not compute aggregates; counts and similar queries must come from the system of record. The model’s count is unreliable because the retrieval and processing pipeline alters the true total before the model sees it.
The author argues that LLMs should not perform arithmetic over a corpus. In a multi-agent document-search setup, the true total (84) is computed by the database, but the model ends up counting a narrowed, surfaced subset (capped at 30, deduplicated, and gated by permissions). Each stage—search, ranking, deduplication, and permission filtering—distorts the count, so the model’s count is a misrepresentation. The result is a plausible but incorrect answer that misleads users. The recommended approach is to have the model refrain from aggregations and instead present the system-of-record result or clearly delegate to the data source for such questions.
Question for the room: What practices have you adopted to ensure your LLMs don’t fabricate aggregates, and how do you surface the true system-of-record count to users?
— via dev.to
Add a comment
0/2000