Beyond the Wrapper: Building a Semantic Cache with pgvector to Cut LLM Costs by 40%
When you build an AI feature, the initial prototype is almost too easy. You wire up an OpenAI API key, write a prompt, and wait for the JSON response.
When you build an AI feature, the initial prototype is almost too easy. You wire up an OpenAI API key, write a prompt, and wait for the JSON response. But when you move to production, reality hits: users ask the same questions using slightly different phrasing, your latency spikes, and repetitive queries create unsustainable inference costs.
While building this Enterprise AI Proxy, a TypeScript gateway routing LLM requests for internal microservices, we hit this exact wall. Here is how we evolved from a naive implementation to building a highly observable semantic caching layer using Node.js and Supabase pgvector. Attempt 1: The Naive Cache (And Why It Failed) My first instinct was traditional software engineering: throw Redis in front of it. We hashed the incoming prompt and cached the LLM response.
This failed almost immediately.
User A asked: "Summarise the lead data." (Cache Miss - API hit)
User B asked: "Summarise lead data, please." (Cache Miss - API hit)
Because LLMs deal with natural language, exact-match string hashing is useless. We needed a cache that understood intent.
Attempt 2: Semantic Caching The architecture needed to change. Instead of hashing strings, we needed to cache vector embeddings.
The constraints:
We couldn't use OpenAI's embedding API for caching. Doing a network call to save a network call defeated the latency goal.
We needed a database that supported rapid vector similarity search.
The Architecture flow
[User Request] │ ▼ [Local Embedding Generator] (Node.js + @xenova/transformers 'all-MiniLM-L6-v2') │ ▼ [Supabase pgvector] (HNSW Index Similarity Search via RPC) │ ├────────────────────────┐ │ │ [Score > 0.95] [Score < 0.95] (Cache Hit) (Cache Miss) │ │ ▼ ▼ [Return Cached] [Call Groq/OpenAI API] [Response] │ ▼ [Store New Embedding] [in Supabase] │ ▼ [Return Response]
By integrating Xenova's @xenova/transformers directly into our Node.js gateway, we generated 384-dimensional embeddings locally, in milliseconds, with zero network latency. When a request came in, we generated the embedding locally, then fired a custom Postgres RPC (match_prompt) to find the closest historical prompt.
The Hard Part: Tradeoffs and Thresholds This is where AI engineering diverges from traditional backend work. How "similar" do two prompts need to be to return a cached response?
If we set the Cosine Similarity threshold to 0.90, we saved a massive amount of tokens. But we introduced a severe semantic mismatch. A prompt asking "What is X?" would return the cached response for "Why is X?", a disastrous UX failure.
If we set it to 0.99, the cache essentially acted like Redis again, missing minor typos and degrading our cache hit rate.
Observability to the Rescue You can't manage what you don't measure. I designed a telemetry_logs schema in Supabase that tracked prompt_tokens, latency_ms, cache_hit_status, and the exact similarity score of every request.
By analysing the telemetry data, I mapped the false-positive rate against the cost savings. The data showed that 0.95 was the golden threshold for our specific embedding model.
The Result By combining local Node.js embeddings, pgvector, and data-driven threshold tuning, we achieved a ~40% reduction in token costs across our microservices, while entirely bypassing external API latency for cached hits.
Building AI tooling isn't just about prompt engineering. It's about treating AI as a systems architecture problem, instrumenting observability, managing tradeoffs, and building infrastructure that scales gracefully.