Hover a node · click to focus · ←/→ steps
Promote only after eval. Canary a traffic slice on K8s; auto-rollback on SLO breach; GitOps is the audit trail.
When you need this
- Prompt edits ship straight to 100% of users
- Model upgrades cause silent quality drops
- Need audit trail of which prompt version served when
- Product wants A/B on system prompts
Prerequisites
- Prompt registry (Langfuse prompts or Git)
- Gateway routing by version tag
- Offline eval suite + online metric
Tools
Self-host for regulated workloads; cloud for faster pilot.
Deploy as single ingress before adding a second LLM vendor.
Run in CI before prompt promotion - pairs with Langfuse datasets.
Steps
- 1
Version prompts in a registry
Every production prompt has an ID + version. Apps request by label (prod, canary), not inline strings.
- 2
Offline gate before canary
Promptfoo/Ragas suite must pass thresholds on the candidate version.
- 3
Split traffic at the gateway
LiteLLM (or app flag) sends N% to canary prompt/model. Tag traces with version for analysis.
- 4
Promote or rollback on metrics
Compare faithfulness, latency, CSAT/task success vs control. Auto-rollback on regression; promote by raising % stepwise.
Adoption pitfalls
- Canary without a control group
- Promoting on latency alone while quality tanks
- Hardcoded prompts still bypassing the registry
Adoption checklist
- No unversioned prompts in prod
- Offline eval required before canary
- Traffic split and rollback automated
- Version visible on every trace
SEER REAL assessment / sprint
Assessment checks release discipline for prompts. Sprint adds prompt registry + gateway canary for one high-traffic surface.