For months I treated embedding search as if it were the whole system: query to vector, cosine similarity against pgvector, top 10, ship it. It worked... until you looked at the ordering of the results closely. The right candidate was almost always there, but in position 4, or 7 — rarely in position 1. The lesson I resisted accepting: embeddings are an excellent coarse filter and a mediocre fine-grained sorter. The reranker is the piece that separates "the relevant results are on the page" from "the best result is first". This article is how I integrated one into the JXBS matching engine without wrecking latency.
Why the bi-encoder falls short
An embedding compresses an entire document into a fixed vector before knowing what you're going to ask it. That's its great advantage (you can precompute and index it) and its structural limit: the compression loses the nuances that only matter against a specific query. Two candidate profiles can be equally "close" to a job opening in vector space for different reasons, and cosine similarity can't tell whether the closeness comes from what's essential or what's incidental.
A reranker is a cross-encoder: it receives the query and the document together, and produces a relevance score by looking at the interaction between the two. It's far more precise at ordering, and far more expensive: you can't precompute anything, every query-document pair is an inference. That's why nobody ranks an entire corpus with a cross-encoder. The pattern is retrieve cheap, reorder expensive over few candidates.
The architecture: retrieve 50, rerank to 10
My pipeline ended up in two stages. The first is the one I already had: pgvector retrieves the 50 nearest neighbors. The second passes those 50 through the reranker and keeps the best 10 according to the new score.
-- Stage 1: cheap retrieval, deliberately over-retrieving
SELECT id, titulo, contenido
FROM documentos
ORDER BY embedding <=> $1
LIMIT 50;
// Stage 2: reranking over the candidates
const scored = await rerank({
query: userQuery,
documents: candidates.map((c) => c.contenido),
});
const top = scored
.sort((a, b) => b.relevanceScore - a.relevanceScore)
.slice(0, 10);
The number 50 isn't sacred. It's the balance I found between two failure modes: retrieve too few and the right document never even reaches the reranker, so there's nothing to reorder; retrieve too many and you pay latency and inference for candidates that never had a chance. I chose 50 because in my tests the correct result was inside the bi-encoder's top 50 practically always; in exchange, I accept that if retrieval fails at the root, the reranker won't rescue it. Garbage in, garbage out — just better sorted.
Latency: the real price and how I contained it
Reranking adds a network call and an inference per request. In my case that meant going from ~80ms to ~350ms at p50. For an interactive search that was too much, and I contained it with three decisions:
First, only rerank when it matters. If the bi-encoder's top candidate score has a wide lead over the second, the ordering is already clear and I skip the second stage. Second, truncate documents: the reranker doesn't need the document's 3,000 words — the title and the first relevant fragment are enough to discriminate, and cost grows with tokens. Third, cache normalized query-document pairs; in a real product, queries repeat far more than you'd think.
💡 The reranker doesn't improve your retrieval: it improves your precision at the top. If your problem is that good results don't even show up in the top 50, your problem is in the embeddings or the chunking, and no reranker will save you.
How I measured it was worth it
Before integrating anything I built a small evaluation set: ~100 real queries with the result a human considered correct marked by hand. The metric I cared about was MRR (mean reciprocal rank): if the correct result is first it scores 1, second scores 0.5, and so on. With the bi-encoder alone I had a mediocre MRR; with the reranker it went up clearly and consistently. I'm not giving exact figures because they depend brutally on the domain and the dataset, but the pattern repeats on almost any corpus: the bi-encoder finds, the cross-encoder orders.
What does generalize is the method: don't integrate a reranker because it's fashionable. Build the 100 human-judged queries first, measure your baseline, and let the number make the decision. In my case the number was conclusive and the extra latency was paid for.
Trade-offs I accepted
I chose a managed reranker via API instead of serving my own cross-encoder, because at my scale the per-request cost was lower than the fixed cost of keeping a GPU warm. In exchange, I accepted an external dependency in the search's critical path, mitigated with a fallback: if the reranker doesn't respond within 500ms, I return the bi-encoder's ordering as-is. Slightly worse search is infinitely better than search that's down.
I also accepted that the system is harder to reason about: there are now two models that can degrade independently. That's why the evals run against the full pipeline, not against each piece in isolation. What reaches the user is the output of the whole chain, and that's the only thing worth measuring.
