Caching

CAELUM can store the answers to requests your application has already made and serve them again instantly, instead of calling (and paying) the provider a second time.

Exact cache

from caelum import OpenAI

client = OpenAI(cache=True)

client.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": "Hi"}])
# Same request again: served from the local cache, no API call, no cost.
client.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": "Hi"}])

Check whether a response came from the cache:

import caelum

response = client.chat.completions.create(...)
caelum.cache_hit(response)   # True / False
caelum.cache_type(response)  # "exact", "semantic" or None

What counts as "the same request"

The cache key is a SHA-256 hash of every parameter that can change the answer: model, messages, temperature, top_p, max_tokens, tools, tool_choice, response_format, seed, stop, and any other generation parameter.

Only transport parameters are ignored: timeout, extra_headers, user, metadata, store, stream_options, safety_identifier and prompt-caching hints.

Requests sent to different servers (base_url) or with different credentials (API key, organization, project) never share answers: in a multi-tenant app, one customer's key is never served another customer's cached answer. Only a short hash of the key is used to tell them apart; the key itself is never written to the cache file. Rotating a key therefore starts with an empty cache for the new key.

What is never cached

• Streaming requests (stream=True).
• Requests with tool_choice="required".
• Failed requests (errors are never stored).
• Structured outputs through chat.completions.parse(...): the call is forwarded to the SDK untouched (use create with response_format to get caching).

Options

client = OpenAI(cache=True, cache_ttl=3600, cache_path="./my-cache.sqlite")

Storage and privacy

• The cache is a local SQLite file on your machine. Nothing is sent anywhere.
• Values are stored as JSON — never pickle, which could execute code from a tampered file.
• On Linux and macOS the cache folder is created readable by you only (0700, file 0600): it contains your prompts and answers, which may include personal data. An existing folder or file keeps its permissions.
• Safe with forking servers (gunicorn, uwsgi, multiprocessing): each worker process opens its own connections to the cache file.
• The cache never breaks your application: if the cache file is unreadable, read-only, corrupted or the disk is full, CAELUM logs a warning, disables the cache and calls the API normally.

Performance

Measured on 1,000 requests with 30% duplicates (see benchmarks/results/):

• 30% of API calls saved (cost saved scales with your duplicate rate).
• Under 1 ms p95 overhead on cache misses (measured +0.25 to +0.56 ms; target: < 5 ms).
• ~0.1 ms per cache hit, versus hundreds of milliseconds for a real API call.

Semantic cache

Reuse answers to questions that are phrased differently but mean the same thing:

import caelum
from caelum import OpenAI

client = OpenAI(semantic_cache=True)

client.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": "How do I return an item?"}])
# Same meaning, different words: served from the semantic cache.
response = client.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": "How can I send back a product?"}])
caelum.cache_type(response)  # "semantic"

How it decides

1. The last user message is turned into an embedding.
2. CAELUM finds the closest previous question with a cosine similarity of at least semantic_threshold.
3. A verifier (a small, cheap model) checks that both questions ask for exactly the same information before the answer is reused. It is asked in both orders and must say YES twice. If it says NO, fails, or times out, the request simply goes to the API.
4. Only questions asked in exactly the same context are compared: same model, system prompt, conversation history, tools and generation parameters. Only the wording of the last question may differ.

Order of lookups: exact cache → semantic cache → API. A semantic hit is also saved in the exact cache (when cache=True), so the same wording is answered next time without even computing an embedding.

Why a verifier

Embedding similarity alone cannot tell "same meaning" from "opposite meaning": "Convert 10 miles to kilometers" and "Convert 10 kilometers to miles" score 0.95, while "Do you offer refunds?" and "Can I get my money back?" score only 0.57. Measured on 50 paraphrase pairs and 50 look-alike pairs (English and French, benchmarks/scripts/bench_semantic_verifier.py):

The verifier was also tested in realistic conditions (each new question compared with all 100 stored questions): no wrong answer was served. End to end through the real OpenAI API (one shared cache, all 100 pairs): 37/50 paraphrases reused, 0/50 wrong answers, versus 0/50 reused with embeddings alone. Five prompt-injection attempts (a stored question telling the judge to reply YES) were all rejected by the default verifier. These figures come from our calibration set; validate on your own traffic before relying on them.

Cost and latency

• A verification is only made when a candidate is found (similarity at least 0.60).
• A few hundred input tokens on gpt-4o-mini: about $0.00006 per check (measured: $0.0071 for 128 checks in our real-API test). Tracked in client.metrics (verifier_cost_usd, semantic_verifications, semantic_rejections).
• Latency: one or two short calls (a few hundred milliseconds each) in sync code, run concurrently in async code. A cache hit still saves the full answer generation.

Multi-user apps: read this first

The exact cache only reuses an answer for the same request (same system prompt, history and parameters), so it is safe to share between users of the same app.

The semantic cache reuses the answer of another user's similar question. Questions are written by end users, so a malicious user could try to phrase a question that tricks the verifier into accepting a match, and get their (manipulated) answer served to others. CAELUM passes each question to the verifier as escaped data and tells it to ignore any instruction inside, but no LLM judge is perfectly immune to prompt injection. If your users are untrusted and answers could be harmful when reused:

• keep the semantic cache off (semantic_cache=False, the default), or
• separate users, for example with one cache_path per user or per tenant, or by putting the user or tenant in the system prompt (different context = never shared).

Never semantically cached

• Streaming requests and tool_choice="required".
• Messages containing images, audio, files or tool results (text-only questions only).
• The embeddings endpoint (returning the vector of a similar text would be wrong).

Options

Choosing the verifier

from caelum import LLMVerifier, OpenAI

# Another OpenAI-compatible judge, e.g. through OpenRouter:
verifier = LLMVerifier(
    base_url="https://openrouter.ai/api/v1",
    api_key="sk-or-...",
    model="mistralai/mistral-small-24b-instruct-2501",
)
client = OpenAI(semantic_cache=True, semantic_verifier=verifier)

If your OpenAI client points to another provider (base_url), pass a verifier explicitly: the default model name gpt-4o-mini may not exist there (verification would then fail safely and nothing would be reused). Small instruction models work best; reasoning models are slow and may return nothing within the one-word budget.

Anthropic and other providers

Anthropic has no embeddings API, so an embedder must be passed explicitly (the question text is then sent to OpenAI for the embedding). There is no default verifier either: pass one, or the strict 0.97 threshold is used.

from caelum import Anthropic, LLMVerifier, OpenAIEmbedder

client = Anthropic(
    semantic_cache=True,
    semantic_embedder=OpenAIEmbedder(api_key="sk-..."),
    semantic_verifier=LLMVerifier(api_key="sk-..."),
)

Any object with a model property and an embed(text) method (and optionally an async aembed(text)) returning caelum.cache.EmbeddingResult can be used as an embedder.

Performance

• Up to 10,000 embeddings are kept (oldest evicted first).
• Search is restricted to the matching context, then scans candidates in memory. For large caches, install the fast extra (pip install "caelum-ai[fast]") to use numpy.
• The embedding call (and the verification, when a candidate is found) adds network latency on requests that go through the semantic cache. This is the price of opting in; the default path (no semantic cache) is unaffected.
• Cache files written by older versions keep working, but their semantic entries have no stored question text, so they are never reused when a verifier is on.

Inspecting the cache

caelum cache stats                      # default file: ~/.caelum/cache.sqlite
caelum cache stats --path my-cache.sqlite --json
python -m caelum cache stats            # same thing, without the console script

CAELUM cache
  File:              /home/me/.caelum/cache.sqlite
  Size on disk:      2.4 MB
  Exact entries:     1250 (40 expired)
  Semantic entries:  310 (0 expired)
  Live entries:      1520
  Oldest / newest:   2026-10-30 09:12 / 2026-11-02 17:10
  Value per reuse:   $ 1.8423  (estimated cost of one more call for every live entry)

  By model:
    gpt-4o-mini-2024-07-18     1100 entries  $ 0.2310
    claude-sonnet-5-5           420 entries  $ 1.6113

The command never changes your cached entries (SQLite may create its temporary -wal/-shm files next to the cache, which are counted in "Size on disk"). "Value per reuse" is what serving every live entry once more would save, from the token counts stored with each answer and the prices in caelum.pricing. Total savings over time are not stored in the file: they are counted per client, in client.metrics.