When an agent goes wrong, the final answer is not enough
A twenty-step agent can fail long before its last sentence. Traces of its choices, tools, and retries are how you find the real mistake.
A final answer cannot explain why an agent chose the wrong tool, repeated a call, or overspent. The useful unit of observation is each model and tool call inside a task trace.
Build one task trace
Give each user task a trace ID and represent model, retrieval, and tool calls as spans. Record model, latency, tokens, tool, status, retries, and error class. OpenTelemetry provides GenAI semantic conventions for consistent fields.
Separate content from metadata
Separate content from metadata. Log lengths, hashes, error classes, and evaluation results by default, not full prompts or personal data. Content sampling needs redaction, restricted access, and short retention.
Alert on user outcomes
Alert on user-level outcomes, not only API errors: task success, cost per task, tool loops, human takeover rate, and quality regressions for specific input groups.
Pre-release checks
- A failed task can be traced to a specific call
- Logs exclude unauthorized sensitive content
- Cost and quality can be sliced by feature, version, and model
Sources
Related
Writing, extraction, long reasoning, and high-volume support do not need the same model. Cost, latency, privacy, and reliability rarely point to one name.
GPUs, operations, inference tuning, and downtime all land on the same ledger. Self-hosting only makes sense after you have done the math on volume and people.