The Bretton Evaluator in action: Agents V2 vs. Agent V1

‍The last part of our series on running AI agents in production inside a financial institution. The second part about our architecture described how we moved from Agent V1 to Agents V2, and why. This one shows the difference: two due-diligence agents run through both generations on the same subjects, graded by the same independent judge, and read defect by defect.

Before we trust our products, we rigorously test them

As we discussed in this blog about rebuilding our underlying architecture, we recently rebuilt the architecture that executes our agents. Agents V2 orchestrates an investigation across a roster of research investigators, each owning the related work for one entity, instead of running it in parallel. Everyone on the engineering team could immediately feel the improvement. Outputs read more clearly and demos went extraordinarily well. But before we moved any customers on to the new architecture, our Applied AI team ran two full benchmarks through both versions, subject by subject, to ensure we had stringent evidence of improvement.

We graded every V1 and V2 run with the Bretton Evaluator, our independent LLM judge, and V2 improved on every measure we track. On enhanced due diligence for individuals, errors fell 37%, and the share of reports with a perfect score rose 17 points. On company due diligence, errors fell 23% and that share rose 12 points. Across the two, V2 cut the errors V1 carried on the same subjects by 30%, and neither gained any run time. In this post, you'll learn how we established that, and what the new architecture specifically fixed.

Both benchmarks improved on all four dimensions

On KYC, V2, the mean score improved from 0.82 to 0.88. Claim support improved the most with a 28.8 point jump. On KYB, the mean score climbed from 0.72 to 0.81. Similarly, Claim support jumped the most with a 27.2 point improvement while workflow process nearly matched it with a 26.4 point shift.

Because every subject is compared against itself rather than against a batch average, a gain cannot be an artifact of one version drawing easier cases: the cases are identical, the skills are identical, and the only thing that differs is the architecture executing them.

The experiment: comparing two agent generations

We ran the benchmark on two kinds of work. Know Your Customer (KYC) is enhanced due diligence on an individual through seven checks, such as address verification, source of wealth and sanctions screening. Know Your Business (KYB) is due diligence on a company through thirteen, including the owners and directors behind it. Each has 250 subjects, hidden from the agents: half fictional with document packs and planted defects, half real companies and public figures described only by public records, and a handful of name-only plants that exist nowhere, which tests restraint.

V1 and V2 are copies of the same agent with the same skills. Only the architecture underneath differs.

What a score actually measures

The rubric scores four things, each pass or fail:

• Claim support: is every claim the report makes actually supported by the evidence it cites?
• Workflow process: did it follow the steps its instructions prescribe?
• Investigative completeness: did it pursue the leads that mattered?
• Deliverable fitness: is the output shaped, structured and cited the way the template requires?

Four pass-or-fail bars make a coarse instrument, and we treat it that way. A perfect score means a run cleared every bar. The real signal is in the findings themselves, so we read the findings and use the score as a sort order.

The design decisions behind the numbers

Reading the judge's findings across both benchmarks, the gains line up with the design decisions described in the architecture piece.

Process discipline. In V2 an agent is no longer one large instruction blob. It is compiled from layers: doctrine, reusable technique, the customer's workflow and governed reference data. The platform decides deterministically which investigators exist for a given subject and what each must resolve before its work is accepted. Skills run only against the right subject, mandated lookups run, and a report describes only work that was actually done. Prompting is not enforcement. Structure is.

Identity before attribution. Extraction and entity resolution used to be setup work. In V2 identity is a first-class investigator: it establishes who the case is actually about and which owners, directors and counterparties matter, and everything downstream inherits that roster. A screening hit becomes a confirmed match only when a durable identifier ties it to the subject, and on a company file clearance never extends to a principal who was not screened. That is the failure this rebuild was built to remove, and it is where KYB gained most.

Coherent reports. V2 treats findings as explicit artifacts and reconciles them before the narrative is written, pushing each conflict back to the investigator that already holds the evidence. The summary line follows the findings beneath it, and sections of one report agree with each other. Coherence is an architectural property, and this is what it looks like when you measure it.

Grounded detail. Evidence lives outside the prompt in durable case state, and a citation counts only when the harness knows the source was actually retrieved. Details trace back to tool output, and planted document defects are surfaced rather than passed over. That is the mechanism behind the claim-support gain on both benchmarks.

The standard we hold ourselves to

An eval that produces a number tells you whether to ship. An eval that produces findings tells you what to build next. Running two workloads rather than one is what separates improvements that belong to the architecture from ones that belong to the workload.

The method is the part worth copying: run the same work twice, change one thing, grade the whole trace, and read the findings rather than the score. It is slower than a demo, and it is the only version a model risk team can act on.

The architecture piece made the argument. This is the evidence, and we will keep publishing it as the system changes. If you want to see how we evaluate agents before they reach a customer's queue, we should talk.