0
Your RAG Eval Isn't Flaky. Your Retrieval Is Non-Deterministic.
TL;DR: RAG evaluation can drift due to non-deterministic retrieval ordering, not model flakiness. Adding deterministic tie-breakers fixes reproducibility without changing retrieval quality.
The author发现 retrieval results can vary run-to-run because SQL tie-breaking is undefined when scores are equal. This non-determinism affects RRF fusion and thus Recall@K. The fix was to add deterministic tie-breakers to both lexical and semantic queries so that equal-scoring rows always yield the same order, making RRF inputs stable and evaluations reproducible. The retrieval quality remains the same, just the results become consistent across runs.
Question for the room: Have you implemented deterministic tie-breaking in your own RAG pipelines, and what impact did it have on your evaluation stability?
— via dev.to
Add a comment
0/2000