Your LLM bill in production doesn't grow because of hard questions: it grows because of repeated ones. In any product with an assistant, a huge share of the traffic is variations of the same queries — "how do I change my password?", "how to reset my password", "forgot my password". A traditional exact-key cache captures none of that, because the text is never identical. A semantic cache does: it turns the question into an embedding, checks whether you already answered something similar enough, and if the nearest neighbor clears a similarity threshold, it returns the stored answer without touching the model. I chose this pattern in a real product because cost per request mattered more than absolute freshness of every answer; in exchange, I accepted the complexity of managing thresholds and invalidating entries. This article is what I learned.
The mechanics: pgvector is enough
You don't need new infrastructure. If you already use PostgreSQL with pgvector for semantic search — which is my usual setup with Prisma — the cache is just one more table:
CREATE TABLE llm_cache (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
query_text TEXT NOT NULL,
query_embedding vector(1536) NOT NULL,
response TEXT NOT NULL,
model TEXT NOT NULL,
hit_count INT DEFAULT 0,
created_at TIMESTAMPTZ DEFAULT now(),
expires_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX ON llm_cache
USING hnsw (query_embedding vector_cosine_ops);
The read path is a nearest-neighbor query with a threshold:
const [hit] = await prisma.$queryRaw<CacheHit[]>`
SELECT response, 1 - (query_embedding <=> ${embedding}::vector) AS similarity
FROM llm_cache
WHERE expires_at > now()
ORDER BY query_embedding <=> ${embedding}::vector
LIMIT 1
`;
if (hit && hit.similarity >= 0.92) {
return hit.response; // cache hit: zero tokens spent
}
If there's no hit, you call the model, store the response with its embedding, and the next user asking the same thing in different words no longer pays.
The threshold is a product decision, not a technical detail
The whole pattern lives or dies on that 0.92. Too low, and you serve wrong answers to questions that only look alike on the surface — "how do I cancel my subscription?" and "how do I cancel a payment?" have uncomfortably close embeddings. Too high, and your hit rate collapses and the cache never pays for its own complexity.
My approach: start at 0.95, log every hit with both the original and the cached question, and manually review a sample every week. Lower the threshold only when the false positives in the next band are acceptable. In domains with a closed vocabulary (support for a specific product) I've gone down to 0.90; in open domains I wouldn't go below 0.93.
💡 The similarity threshold is not a hyperparameter you tune once: it's a product policy defining how much imprecision you tolerate in exchange for cost. Treat it as such — with logging, periodic review, and an owner.
Invalidation: the hidden price
A semantic cache inherits the classic cache problem and adds one of its own. The classic one: answers expire. If your product changes its billing flow, every cached answer about billing is now a well-written lie. That's why every entry carries expires_at and, more importantly, a category-based invalidation mechanism: I tag entries with the functional area they touch, and when I deploy a change in that area, I wipe the whole category. It's blunt, but it's predictable.
The problem of its own: personalized answers. If the model's response includes user data ("your current plan is Pro"), caching it and serving it to another user is a data leak, not an optimization. My rule is binary: only what passes a prior "generic question" classifier enters the cache. Anything that depends on user context stays out, no exceptions and no special cases.
When not to do it
Trade-offs rule. A semantic cache pays off when traffic has high semantic redundancy (support, FAQs, onboarding), cost per call is relevant, and latency matters — a hit responds in ~50ms versus the 2-4 seconds of a large model. It doesn't pay off when every question is unique (creative tools, analysis of user documents), when answers depend on personal context, or when volume is so low that the LLM bill is noise.
In my case, the signal to build it came from looking at the logs: when you see the same intent expressed twenty different ways day after day, the cache justifies itself. If you haven't looked at your logs yet, start there — you may not need any of this, and that's good news too.
