Here is exactly how we audit your agent. Steal it.

Is the agent reading live data or guessing? Stale snapshots, broken indexes, and mis-tuned similarity search are the highest-frequency source of hallucinated tool arguments and outputs.

• Live data source connected (not a stale export)
• Retrieval freshness within SLA
• Embedding model and index version pinned
• Reranking on or off justified by an offline eval
• Citations or source references surfaced in trajectories

What is the agent allowed to do and forbidden from doing on each step. Explicit scope and exit conditions are what prevent planner loops and silent drift.

• Hard scope limits encoded in the planner
• Max steps and loop detection enforced
• Failure mode examples in few shot
• Planner versioning and rollback in place
• Refusal trajectories tested for edge cases

When the agent calls a tool to act (book, charge, send, write, query), is it calling the right tool with the right arguments, every time.

• Tool schemas strict and validated server side
• Idempotency keys on side effecting tools
• Tool-call accuracy at or above 95 percent on eval set
• Argument validation before execution
• Graceful failure on tool errors with bounded retries

Does the agent remember the right things, forget the right things, and isolate across tenants. Bad memory causes repeated questions, leaked context, and cross client bleed.

• Memory scope per user, per session, per thread
• PII redaction in long term memory
• Memory eviction policy documented and tested
• Cross session and cross tenant leakage tested
• Handoff to human preserves planner context

Can you measure whether the agent is getting better or worse week over week. Without offline and online evals you are flying blind.

• Golden set of 50 or more representative trajectories
• Trajectory match scoring, not just final answer
• Regression suite runs on every deploy
• LLM as judge calibrated against humans
• Domain specific metrics (not just BLEU or BERT score)

When something goes wrong in production, can you find out within minutes, not weeks. Traces and quality scores on live traffic, not after the invoice arrives.

• Every trajectory traced end to end (OTel or vendor)
• Quality scores running on live traffic
• Alerts on hallucination, refusal, and latency drift
• Dashboards usable by non technical stakeholders
• Incident playbook documented

Are you paying for value or paying for retries on broken trajectories. Most production agents leak 40 to 60 percent of token spend on planner loops and tool retries.

• Cost per successful task tracked
• Model routing (cheap to expensive) where applicable
• Prompt compression on long context windows
• Retry and timeout budgets enforced with circuit breakers
• Spending alerts before invoice surprises

What is the worst trajectory the agent can take. Have you tested for it. Prompt injection through tool outputs is now the dominant attack surface.

• Tool output prompt injection defenses in place
• Jailbreak and adversarial input resistance tested
• Industry compliance checks (Fair Housing, HIPAA, FINRA, GDPR)
• Toxic or off brand output filters
• Human in the loop on high stakes tool calls