Hover a node · click to focus · ←/→ steps
Ingest → embed → index → retrieve → generate. Offline + online eval gates block regressions before promote.
When you need this
- Langfuse alone is not enough for notebook-to-prod debugging
- Need local/OSS tracing before enterprise APM spend
- Retrieval quality opaque in production traces
- Want span-level eval overlays
Prerequisites
- App or gateway you can instrument
- Phoenix deployment (local or hosted)
- PII redaction policy for stored spans
Tools
Strong OSS alternative or complement to Langfuse; evaluate self-host ops.
Instrument gateway and app tier first; expand to training jobs later.
Wire into CI with a fixed golden set - do not rely on LLM-judge alone without human calibration.
Steps
- 1
Instrument the critical path
Trace LLM calls, retrieval, and tool use. Propagate IDs through the gateway. Redact PII before export.
- 2
Stand up Phoenix
Run Phoenix for the team; import traces. Confirm engineers can open a failing user session end-to-end.
- 3
Overlay evals
Attach Ragas/faithfulness scores to spans. Filter traces by low scores for triage queues.
- 4
Feed the release loop
Sample failing online traces into the golden set monthly so offline evals track real failures.
Adoption pitfalls
- Storing raw PII in traces
- Instrumenting everything with no sampling → cost and noise
- Traces without product/team tags
Adoption checklist
- Critical LLM path fully traced
- PII policy applied to spans
- Low-score triage queue owned
- Monthly golden-set refresh from prod failures
SEER REAL assessment / sprint
Assessment checks debug time for a recent LLM incident. Sprint instruments one path into Phoenix with eval overlays and a triage ritual.