Agentic Systems: Trust Is a Case You Build

In a quarterly risk review, the Chief Risk Officer asks the obvious question: “How do we know the agent is making the right decisions?” The agent adjudicates claims - what a damaged roof is worth, whether a story holds together, when to pay. The honest complication: hand the same claim to three veteran adjusters and you get three defensible numbers, none “correct.” The benchmark the agent would be graded against doesn’t exist, and never did - not for the agent, and not for the humans who did the job before it.

And the difficulty isn’t only that the right answer is unknown - the agent itself can’t be exhaustively tested the way a workflow can. A deterministic workflow has a finite set of branches: every path can be enumerated and checked until coverage is complete. An agent runs no fixed path. It runs a loop - understand → reason → act → remember, then begins again on what it just learned. Each turn is state-dependent, so the same starting input can take a different route depending on what the agent has seen and stored in memory. The path space isn’t a tree you can walk; it is combinatorial and effectively unbounded. You cannot pre-enumerate and exhaustively cover every path the way you would a workflow, so coverage stops being the right yardstick. The testing doesn’t shrink - it changes shape: continuous, statistical, and running in production, rather than a fixed suite you run once and tick off.

Between them, these two facts dismantle the usual playbook. The agent didn’t fail your benchmark - it exposed that you never had one; the disagreement was always there, hidden inside a workforce nobody ever measured this precisely. You cannot prove an agent correct when correctness isn’t a single value, and you cannot certify it by coverage when its paths can’t be enumerated. Neither limit lowers the bar - it raises it. Rather than one assertion or one coverage run, you build layered, instrumented evidence that its judgment is trustworthy within defined boundaries, and you keep gathering it while the system is live. Two artifacts carry that evidence: the reasoning trace (how a single decision was made) and the confidence map (where the agent can be trusted and where it stops). The five layers below produce and validate those two artifacts.

Layer 1 - Constraint adherence

Instrument Versioned pass/fail rules, evaluated on every decision before it ships.

This is the part of the task where ground truth does exist: authority limits, mandatory disclosures, fraud escalation. Binary, non-negotiable, no judgment involved. A usually-sensible model can still exceed its authority under the right prompt; a config rule catches that deterministically, before a regulator does.

constraints:
  - id: authority_limit
    rule: settlement_amount <= adjuster_authority[agent_tier]
    on_violation: block_and_route_to_human
    severity: critical

  - id: regulatory_disclosure
    rule: |
      if claim.state in MANDATORY_DISCLOSURE_STATES
      then disclosure_notice.attached == true
    on_violation: block
    severity: critical

  - id: fraud_escalation
    rule: |
      if fraud_signal_score >= 0.80
      then escalation.flag == true and payment.hold == true
    on_violation: block_and_alert_SIU
    severity: critical

Anything that fails here never reaches the layers where judgment is actually tested.

Layer 2 - Reasoning quality (produces the trace)

Instrument A structured reasoning trace per decision, scored by an LLM-as-judge at scale and calibrated against human review.

Every decision emits the factors weighed, the range considered, a calibrated confidence score, and a dissent check - the agent’s argument against its own conclusion before committing.

DECISION: Settle water-damage claim at $14,200

FACTORS:
  + Damage pattern consistent, kitchen + adjacent wall
  + Plumber's report corroborates supply-line failure (covered)
  + Claimant history clean, 8 yrs, no prior water claims
  - Moisture documentation incomplete for subfloor
  - Contractor estimate 12% above regional median

RANGE ASSESSED: $11,800 - $15,600
CONFIDENCE: 0.81 (lowered: subfloor documentation gap)

DISSENT CHECK:
  Strongest case against: subfloor moisture unverified; if
  pre-existing, scope shrinks ~$2,400. Recommend holding the
  subfloor line pending one more reading rather than denying -
  evidence supports partial deferral, not denial.

The dissent check separates reasoning from fluent confidence: a system that can’t argue against itself is narrating a foregone conclusion, not reasoning. The LLM-as-judge handles volume; a weekly human-scored sample keeps the judge calibrated. This is interpretability in usable form - the actual chain of judgment, legible to whoever has to defend it. Its limit: a trace explains one decision and says nothing about the thousand around it. That requires the next layer.

Layer 3 - Outcome patterns and drift

Instrument Distribution monitoring across thousands of decisions, a curated regression suite, and unannounced live test cases.

Some failures exist only in aggregate. Rolling averages by claim type, Kolmogorov-Smirnov tests against a baseline distribution, and reopen rates surface them.

Worked example: mold-remediation claims pass every individual trace for weeks - reasoning sound, constraints honored, confidence appropriate - while the rolling distribution drifts low. The agent undervalues remediation scope by a few hundred dollars consistently, traced to a stale cost reference. Each decision is individually defensible; the error lives in the pattern, and the KS test flags it before reopen rates do. No human reading claims one at a time catches this.

Underneath sits a regression suite of 4,000+ real cases, each with a known-good band, run on every model change. It is the institutional memory that stops the system re-learning lessons it already paid for.

The regression suite has one blind spot: the agent is, in effect, taking a known exam. So you also run the equivalent of mystery shopping - synthetic but realistic cases, with known-good bands, injected into the live decision stream unannounced and indistinguishable from real traffic. This catches a failure mode unique to systems capable enough to have a context: behaving differently when they can tell they’re being evaluated. A retailer plants secret shoppers because staff act differently when observed; the same precaution applies the moment an agent is good enough to notice the difference. The canary it can’t see is the only honest test of how it behaves unwatched.

Layer 4 - Comparative standing

Instrument Blind expert panels, scoring decisions without knowing which are AI and which are human.

This is how you construct a benchmark in a domain that doesn’t ship one. Three findings:

• Experts agree with each other ~74% of the time - ~28% disagreement among the very people treated as ground truth. The baseline you would grade the agent against is itself uncertain by a quarter, which quietly invalidates every accuracy figure ever reported in this kind of work.
• When reviewers can’t see the label, a well-built agent’s decisions do not score systematically below the humans’.
• And the inverse of the 74% is worth measuring directly: re-run the same case with minor rephrasings and check how often the agent disagrees with itself. Humans disagree with each other ~28% of the time; an agent whose self-disagreement runs well below that isn’t merely as good as the workforce - it is more internally consistent than it.

The measurable question is therefore not “does the agent match the correct answer,” but “is its judgment indistinguishable from a competent professional’s to experts who can’t see the label.” That has an answer.

Layer 5 - Boundaries of competence (produces the confidence map)

Instrument Adversarial testing that maps where the agent’s confidence collapses and it escalates.

You steer deliberately into hard cases - concurrent causation, emotional distress, novel ambiguity - and measure not whether the agent is right (often there is no right) but whether it knows it is out of its depth and hands off.

A wrong agent is a problem; a wrong and confident agent is a catastrophe. The map is arguably worth more than the decisions it bounds: a regulator doesn’t need brilliance everywhere, only a reliable record of where the agent isn’t, and proof that it withdraws there.

Back to the boardroom

The answer to the CRO’s question is three instruments, not a single number:

• Performance - traces showing how each decision was reached, and blind panels showing the agent holds up against experienced adjusters without labels.
• Observability - live distribution monitoring that surfaces drift in days, not quarters.
• A brake - hard constraints blocking rule breaches before they ship, and a confidence map showing the agent escalating exactly where its competence ends.

“So you can’t tell me it’s always right.” No - right was never a single place, for the agent or the adjuster. But you can show measured performance, fast-visible drift, and reliable withdrawal at the boundary. That is a stronger basis for trust than an accuracy score, and more than was ever asked of the humans doing the work.

Stop asking whether the agent is right. Ask the question that has an answer:

Do we have sufficient evidence to trust its judgment, within boundaries we have defined?