The Bretton Evaluator in action: Agents V2 vs. Agent V1

‍The last part of our series on running AI agents in production inside a financial institution. The second part about our architecture described how we moved from Agent V1 to Agents V2, and why. This one shows the difference: two due-diligence agents run through both generations on the same subjects, graded by the same independent judge, and read defect by defect.

Before we trust our products, we rigorously test them

As we discussed in this blog about rebuilding our underlying architecture, we recently rebuilt the architecture that executes our agents. Agents V2 orchestrates an investigation across a roster of research investigators, each owning the related work for one entity, instead of running it in parallel. Everyone on the engineering team could immediately feel the improvement. Outputs read more clearly and demos went extraordinarily well. But before we moved any customers on to the new architecture, our Applied AI team ran two full benchmarks through both versions, subject by subject, to ensure we had stringent evidence of improvement.

We graded every V1 and V2 run with the Bretton Evaluator, our independent LLM judge, and V2 improved on every measure we track. On enhanced due diligence for individuals, errors fell 37%, and the share of reports with a perfect score rose from 42% to 59%. On company due diligence, errors fell 23% and that share rose from 26% to 38%. Across the two, V2 cut the errors V1 carried on the same subjects by 30%, and neither gain cost any run time. In this post, you'll learn how we established that, and what the new architecture specifically fixed.

The experiment: comparing two agent generations

We ran this benchmark twice, on two different kinds of work. Know Your Customer (KYC) is enhanced due diligence on an individual: 250 subjects, each run through seven checks. A check is one verification step an analyst would otherwise perform by hand: address verification, source of wealth, sanctions screening, political exposure, adverse media, geographic risk and industry risk. Know Your Business (KYB) is due diligence on a company: 250 subjects and thirteen checks, which add registration verification, document analysis, company source of wealth, online reputation, domain history, and a review of the company’s website and terms. Each company comes with staged owners and directors behind it, so the agent has to piece together an ownership structure and screen the people inside it, not just the entity.

Both benchmarks mix three kinds of subjects, and this is hidden from the agents. Half are fictional, with a full document pack and planted defects. Half are real companies and real public figures described only by sourced public records, which tests research and screening against genuine hits. A handful are name-only plants that exist nowhere, which tests restraint: the right answer is to find nothing and say so.

In each benchmark, both v1 and v2 are copies of the same source agent with the same skills enabled, in the same organization and environment. The only difference is the architecture underneath. Every run is then graded by the Bretton Evaluator, which reads the entire trace, every tool call, every intermediate conclusion and the final report, against a four-dimension rubric, and writes up each defect it can prove with the causal chain behind it.

250 KYC subjects and 250 KYB subjects carry a verdict in both v1 and v2, and every figure below is a paired comparison of a subject against itself rather than an average over two different populations. V1 and V2 are graded by the same judge against the rubric for its architecture, so the two are read as paired defect counts on the same subjects.

What a score actually measures

The rubric scores four things, each pass or fail:

• Claim support: is every claim the report makes actually supported by the evidence it cites?
• Workflow process: did it follow the steps its instructions prescribe?
• Investigative completeness: did it pursue the leads that mattered?
• Deliverable fitness: is the output shaped, structured and cited the way the template requires?

Four pass-or-fail bars make a coarse instrument, and we treat it that way. A perfect score means a run cleared every bar. The real signal is in the findings themselves, so we read the findings and use the score as a sort order.

Both benchmarks improved on all four dimensions

On KYC, V2 produced 37% fewer findings than V1 on the same 250 subjects. Perfect-score runs went from 42% to 59% and the mean score from 0.82 to 0.88. On KYB, findings dropped 23%, perfect runs went from 26% to 38%, and the mean score climbed from 0.72 to 0.81.

Because every subject is compared against itself rather than against a batch average, a gain cannot be an artifact of one version drawing easier cases: the cases are identical, the skills are identical, and the only thing that differs is the architecture executing them. Two dimensions improved most: claim support on KYC and workflow process on KYB.

Claim support, the dimension that asks whether a conclusion is actually supported by what the agent cited, is the hardest test in this rubric. The other three dimensions ask whether the agent behaved correctly: whether it followed its steps, pursued the right leads, and produced the right shape. Claim support asks something it cannot satisfy by compliance alone — whether the source it cited actually substantiates the claim, and about the right entity — which is why a fluent, well-structured report can still fail it, and why it carries the lowest pass rate of the four on both workloads. It is also the failure that travels furthest: a claim the evidence does not support is a false statement presented as fact, not a cosmetic slip. It gained on both workloads: 63.6% to 75.6% on KYC and 50.8% to 57.2% on KYB, a 6.4-point gain on the hardest test of the hardest workload. Workflow process on KYB gained 18.0 points, from 68.0% to 86.0%; on a thirteen-check workflow with an ownership chain to pierce, that is the difference between an agent that mostly follows its instructions and one that reliably does. Investigative completeness gained 8.8 points on KYB, and deliverable fitness reached 98.4% on KYC.

Pass rates describe the population as a whole. The stricter test is the same subject against itself: a report that failed a dimension under V1 and passes it under V2. On KYC, V2 turned 15.1% of all dimension checks from fail to pass: 28.8% of subjects gained claim support, 17.2% workflow process, 8.0% investigative completeness and 6.4% deliverable fitness. On KYB it turned 20.2%: 27.2% of subjects gained claim support, 26.4% workflow process, 20.4% investigative completeness and 6.8% deliverable fitness. Claim support leads on both, and it is the dimension that decides whether an analyst can act on a report without redoing the research.

The design decisions behind the numbers

Reading the judge's findings across both benchmarks, the gains line up with the design decisions described in the architecture piece.

Process discipline. In V2 an agent is no longer one large instruction blob. It is compiled from layers: doctrine, reusable technique, the customer's workflow and governed reference data. The platform decides deterministically which investigators exist for a given subject and what each must resolve before its work is accepted. Skills run only against the right subject, mandated lookups run, and a report describes only work that was actually done. Prompting is not enforcement. Structure is.

Identity before attribution. Extraction and entity resolution used to be setup work. In V2 identity is a first-class investigator: it establishes who the case is actually about and which owners, directors and counterparties matter, and everything downstream inherits that roster. A screening hit becomes a confirmed match only when a durable identifier ties it to the subject, and on a company file clearance never extends to a principal who was not screened. That is the failure this rebuild was built to remove, and it is where KYB gained most.

Coherent reports. V2 treats findings as explicit artifacts and reconciles them before the narrative is written, pushing each conflict back to the investigator that already holds the evidence. The summary line follows the findings beneath it, and sections of one report agree with each other. Coherence is an architectural property, and this is what it looks like when you measure it.

Grounded detail. Evidence lives outside the prompt in durable case state, and a citation counts only when the harness knows the source was actually retrieved. Details trace back to tool output, and planted document defects are surfaced rather than passed over. That is the mechanism behind the claim-support gain on both benchmarks.

The standard we hold ourselves to

An eval that produces a number tells you whether to ship. An eval that produces findings tells you what to build next. Running two workloads rather than one is what separates improvements that belong to the architecture from ones that belong to the workload.

The method is the part worth copying: run the same work twice, change one thing, grade the whole trace, and read the findings rather than the score. It is slower than a demo, and it is the only version a model risk team can act on.

The architecture piece made the argument. This is the evidence, and we will keep publishing it as the system changes. If you want to see how we evaluate agents before they reach a customer's queue, we should talk.