Skip to content
Gatewyse

Cache

The gateway caches completions in two layers. An exact layer in Redis keyed on a hash of the normalised request, and a semantic layer in MongoDB that matches requests which are similar rather than identical.

The Cache page lets you inspect what is stored, remove a single answer, and flush everything.

How a lookup works

  1. Exact. A Redis GET on the scoped key. A hit returns immediately.
  2. Semantic. On a miss, the request text is embedded and compared against stored entries, racing a three-second timeout — a cache that takes longer than the provider would have is not a cache.
  3. Promotion. A semantic hit is written back into Redis under the exact key, so the same question asked twice more is answered from the fast layer.
  4. Otherwise, a miss.

The semantic pass compares against the fifty most recently used candidates for the requested model. Anything outside that window is not considered, which bounds the work done on the request path.

Candidates are narrowed on the model the caller asked for, not the model that answered. Routing may serve a gpt-4o request from a different provider, and a cached answer belongs to the question, not to whoever happened to answer it that time.

The cache key

A SHA-256 of the normalised request: model, message roles and contents, system prompt, temperature, top-p, max tokens, sorted tool names, tool choice, response format and seed.

The key alone is not the partition. The Redis key is:

aigw:cache:<deployment>:<organizationId|->:<departmentId|->:<cacheKey>

A - marks an absent level, so two different scope paths cannot collide. A caller with no organization shares the deployment-wide namespace; an organization’s answers are not served to another organization.

When the cache is skipped entirely

A request bypasses both layers if any of these hold:

  • stream: true
  • temperature > 1.0
  • tools are present
  • n > 1
  • seed is set

Each of these describes a request whose answer is either not reusable or not deterministic enough to be worth reusing.

Threshold and TTL

Both are deployment-level environment variables. Neither is editable from the console.

VariableDefaultMeaning
CACHE_SIMILARITY_THRESHOLD0.96Minimum cosine similarity for a semantic hit, 0–1
CACHE_DEFAULT_TTL_SECONDS86400Entry lifetime, 24 hours

The Redis TTL is written with ±10% jitter, so a burst of entries created together does not expire together and dump the whole load onto providers in one second.

cachingEnabled and cacheTTLSeconds also exist as policy on the deployment and each scope, and feed the per-request TTL.

Embeddings are local

Semantic matching uses an ONNX all-MiniLM-L6-v2 model loaded from disk, with remote model fetching disabled. No text leaves the deployment to be embedded, which is what keeps the semantic cache usable in an air-gapped install.

If the model cannot be loaded, an entry is stored with an empty embedding and is simply never returned by a semantic match. Exact matching is unaffected.

This is not the same embedding stack the document corpus uses — see Documents.

Reading it in the console

The entries table shows one row per stored answer:

ColumnContent
AnswerThe model, the provider that produced it, and a truncated cache key
ReusesHow many times this entry has been served
MatchThe similarity score recorded on it
StoredWhen it was created, and when it was last served
ExpiresRemaining lifetime, or “Expired” / “No expiry”

Only Stored and Expires are sortable — the list endpoint accepts no other sort field. The page says so above the table, and ranks the two derived panels below (most-reused answers, and matches worth reviewing) client-side over a bounded sample of the newest hundred entries, labelled as such.

The Match figure is the similarity of the most recent semantic match against that entry. An entry only ever served by exact hits keeps its initial 1.0, so a column of 100% means “not measured by a semantic comparison”, not “all answers were perfect”.

The detail drawer shows the model, provider, reuse count, match, token split, timestamps, cache key and the stored answer. The request text itself is embedded rather than rendered, so the drawer does not display it.

Flushing

Flush clears the whole deployment. Both layers, every organization, every department, in one sweep — every scope key begins with the deployment identifier, so there is no partial flush.

The console requires you to type the word flush to confirm, and states the consequence: until the cache refills, every request goes to a provider, latency returns to a full round trip, and every one of them is billed.

Deleting a single entry removes both its MongoDB row and its Redis twin. Doing only the first was a real earlier defect — the control read as working while every cached answer was still being served.

Both destructive operations require deployment scope, whatever permissions the caller holds.

Endpoints

MethodPathPermission
GET/api/admin/cache/statscache:read
GET/api/admin/cache/entriescache:read
DELETE/api/admin/cache/entries/:idcache:delete + deployment scope
DELETE/api/admin/cache/flushcache:delete + deployment scope

Note that flush and single-entry delete are DELETE, not POST.

GET /entries accepts page, limit (capped at 100), sort (createdAt or expiresAt) and order. It has no search or filter parameters, and excludes the stored embedding vector from the response.

cache:read and cache:delete are granted to super-admin and deployment-admin only. Organization and department administrators have no cache permission at all, so this page is not reachable for them.