If your RAG system retrieves roughly the right documents but the best one isn't always on top, you probably need a re-ranker. It's one of the highest-return, lowest-effort upgrades in retrieval.
The two-stage idea
Vector search is fast but coarse — it casts a wide net, returning documents that are similar to the query. But "similar" isn't quite "most relevant." A re-ranker adds a second, more careful stage: take the top candidates from vector search (say the top 50) and score each one against the query with a slower but more precise model, then keep the best few.
First stage: cast a wide net, fast. Second stage: examine the catch carefully, slow. You get both speed and precision.
Why it works
The first-stage retriever has to be fast, so it compares pre-computed embeddings — it never looks at the query and document together in depth. A re-ranker (a cross-encoder) does exactly that: it reads the query and each candidate jointly and judges true relevance. It's too slow to run over millions of documents, but perfect for re-scoring a shortlist of 50.
The payoff
Adding a re-ranker often improves answer quality more than swapping to a better embedding model or a bigger LLM — because it fixes the "right documents, wrong order" problem directly, ensuring the most relevant context lands at the top where the model actually uses it. For a modest latency cost on a shortlist, you get materially better retrieval. If your RAG is "close but not quite," a re-ranker is the first thing to try.
