askalf/deja

deja

deja — own your LLM cache. A hybrid exact + semantic caching proxy: repeat and near-repeat queries serve from your cache, not the provider's meter. Part of Own Your Stack — own your AI infrastructure instead of renting it by the token.

"I've seen this query before."

A hybrid exact + semantic LLM caching proxy. Speaks the OpenAI, Anthropic, and Google Gemini wire APIs, so any client adopts it by changing one base URL. Optimizes for cost, latency, and throughput at once.

client ──▶ deja ──(miss)──▶ OpenAI / Anthropic / Gemini
            │
            ├─ L1  exact-match   (Redis / in-mem)   ── µs, zero risk
            ├─ SF  single-flight (coalesce dupes)   ── throughput + rate-limit
            └─ L2  semantic       (vector + guard)  ── cost, behind a guard

Quick start

cp .env.example .env
npm install
npm run dev            # deja listening on :8787  [L1=memory, semantic=off]
npm test               # full battery: spawns stub upstream + 7 instances, runs all 6 suites

Point any OpenAI client at it:

curl http://localhost:8787/v1/chat/completions \
  -H "authorization: Bearer $OPENAI_API_KEY" \
  -H "content-type: application/json" \
  -H "x-cache-mode: exact" \
  -d '{"model":"gpt-4o-mini","temperature":0,"messages":[{"role":"user","content":"capital of France?"}]}'
# repeat the call → response header  X-Cache: HIT-L1

Anthropic works the same on POST /v1/messages (x-api-key + anthropic-version are passed through). POST /v1/embeddings is also cached (exact-match — embeddings are deterministic and repeat constantly).

Google Gemini works on its native POST /v1beta/models/{model}:generateContent (and :streamGenerateContent) — x-goog-api-key / ?key= are passed through. Gemini puts the model and the stream/non-stream choice in the URL rather than the body, so deja canonicalizes the path for keying: a streamed and a non-streamed request for the same