A vector index gets expensive much sooner than people expect. Not because of the embedding model, which already costs pennies, but because of memory: if you store 3072-dimensional vectors in float32, each one takes 12 KB, and a million vectors is 12 GB of raw data alone, before you count the index graph. What I learned building the matching engine at JXBS is that this size is negotiable along two independent axes —how many dimensions you store and at what precision you store each one— and that cutting on both degrades recall far less than intuition suggests, as long as you use the small vector to search and the big one to decide.
The two axes: dimensions and precision
When you want an embedding to take up less space, there are exactly two levers. The first is truncating dimensions: keeping the first N components of the vector instead of all of them. The second is quantizing: lowering the precision of each component, from float32 to int8 or even to a single bit.
These levers multiply. A 1024-dimensional vector in int8 takes 1 KB versus 4 KB for the same vector in float32, and versus 12 KB for the full 3072-dimensional float32 vector. Twelve times less memory. The question isn't whether that saves money —obviously it does— but how much recall you lose along the way.
Truncating dimensions isn't as barbaric as it sounds
Truncating an arbitrary vector destroys information unpredictably: there's no guarantee the first components are the important ones. What changed the landscape is matryoshka-style training, where the model is explicitly trained so that prefixes of the vector are, on their own, useful representations. The first dimensions carry the coarse signal; the last ones refine nuance.
If your model supports this scheme —OpenAI's main models and several open ones do— truncating is literally slicing the array and renormalizing:
function truncate(vector: number[], dims: number): number[] {
const slice = vector.slice(0, dims);
const norm = Math.hypot(...slice);
return slice.map((v) => v / norm); // renormalizing is mandatory
}
That renormalize is not optional. If you use cosine similarity with normalized vectors and truncate without normalizing again, magnitudes stop being comparable and distances get dirty. It's the most common error and the quietest one: nothing blows up, results just get worse.
If your model is not trained with matryoshka, don't truncate. There the honest reduction goes through PCA or a learned projector, and that's one more component to maintain, version, and reindex.
Quantizing: int8 nearly free, binary with caveats
Dropping from float32 to int8 preserves surprisingly high recall —in my tests the loss has been marginal— in exchange for a quarter of the memory. The trick is calibration: you need to know the actual range of your components in order to map it to int8's range, and that range is computed over a representative sample of your corpus, not over the first batch you happen to have on hand.
Binary quantization is another story. You reduce each component to one bit based on its sign, take up 32 times less space, and compare with Hamming distance, which is an XOR and a popcount: brutally fast. But you lose recall visibly. Nobody serious uses binary as the final answer; it's used as a filter.
💡 The right question isn't "what precision do I use?" but "what precision do I use in each phase?". Searching and deciding are different problems and deserve different representations.
The pattern I use: search cheap, decide expensive
Instead of choosing a single format, I store two. A small, quantized vector to traverse the index, and the full float32 vector —or something close— to rerank the few candidates that survive.
The first phase retrieves, say, the 200 nearest candidates using the cheap vector. It's a recall phase: it only has to guarantee the good ones are inside those 200, not that they're well ordered. The second phase takes those 200, computes similarity with the full-precision vector and orders them for real. That second phase touches 200 vectors, not a million, so index memory doesn't constrain it: the full vectors can live on disk or in a regular Postgres table.
-- Phase 1: cheap recall over the quantized vector (indexed)
WITH candidates AS (
SELECT id
FROM documents
ORDER BY embedding_int8 <=> $1::vector
LIMIT 200
)
-- Phase 2: rerank the 200 with the full-precision vector
SELECT d.id, d.title, d.embedding_full <=> $2::vector AS distance
FROM documents d
JOIN candidates c ON c.id = d.id
ORDER BY distance
LIMIT 10;
This is exactly the same idea as putting a reranker on top of semantic search, but one floor down: instead of an expensive model reordering, it's an expensive vector reordering. And you can combine both.
How I decide the cutoff
There's no magic number. What I do is measure. I freeze a set of queries with their known relevant results, define recall@10 with the full float32 vector as the reference, and try configurations: 3072/float32, 1024/float32, 1024/int8, 512/int8, binary+rerank. Each one gives a (recall, memory) pair. With that table in front of you the decision stops being an argument about opinions and becomes an explicit trade-off choice.
My default bias, when the corpus is large: medium dimensions with int8 in the index, full vector stored separately for the second phase. I chose that because the index's memory cost drops nearly an order of magnitude, in exchange for a two-step query instead of one and for keeping two columns in sync when you reindex. That's the real price, and it needs saying: every compression scheme is a pending migration. The day you change embedding models, you reindex both columns.
And a final warning: don't optimize this before you have the problem. With a hundred thousand vectors, full float32 fits in memory without breaking a sweat and you won't notice the difference. Compression is an answer to a concrete constraint, not a virtue in itself.
