Hover a node · click to focus · ←/→ steps
Apps → tagged gateway → AWS / Azure / GCP providers or self-host. Cost tags feed FinOps; sampled traffic closes the eval loop.
When you need this
- High volume of near-identical support questions
- LLM bill dominated by repeated prompts
- Need cache invalidation when knowledge base updates
- Want gateway-level cache, not app-specific hacks
Prerequisites
- Redis (with vector capability or RedisVL)
- LiteLLM or app middleware for cache hook
- TTL and similarity threshold policy
Tools
Trial for cache hit-rate wins; enforce TTL and invalidation when knowledge bases change.
Deploy as single ingress before adding a second LLM vendor.
Steps
- 1
Measure repeat traffic
From gateway logs, estimate % near-duplicate prompts. Cache only where similarity savings exceed quality risk.
- 2
Deploy semantic cache at the gateway
Embed incoming prompt; lookup Redis; return cached completion when cosine similarity ≥ threshold. Tag cache hits for FinOps.
- 3
Set TTL and invalidation
Short TTL for fast-changing knowledge; event-driven purge when RAG index or policy docs update. Never cache personalized/PII-heavy prompts.
- 4
Monitor quality and savings
Dashboard hit rate, $ saved, and spot-check faithfulness on cached answers. Tune threshold — too loose serves wrong answers.
Adoption pitfalls
- Caching prompts that include user-specific data
- No invalidation after KB change → confident wrong answers
- Threshold too low → semantic collisions
Adoption checklist
- Cache bypass for authenticated/personalized prompts
- Invalidation on knowledge updates
- Hit rate and $ saved on FinOps dashboard
- Threshold review after 2 weeks of traffic
SEER REAL assessment / sprint
Assessment finds repeat traffic and missing cache. Sprint enables gateway semantic cache for one FAQ-like product with invalidation hooks.