Automated evals say pass. A controller says why.
An agent can tie out every balance and still book the wrong number. What an expert reviewer catches that an automatic check misses, with one example from a month-end close.
Most agent evaluations ask a narrow question: did the output match what was expected? For a lot of professional work that question is not enough. The output can match every check you wrote and still be wrong in a way that matters to the people who sign the accounts.
The pass that wasn’t
Take a common close task. An agent reconciles intercompany balances between two entities at quarter end. It pulls both trial balances, matches open items by invoice ID, finds a difference on three lines, and proposes an adjusting entry. The balances tie. The entry posts. An automatic check marks the run as a pass.
A controller reading the same run stops at step three. The difference is foreign exchange, and the agent translated the open balances at the rate on each invoice date. Monetary balances in a foreign currency are carried at the closing rate at the reporting date under both IFRS (IAS 21) and US GAAP (ASC 830). Everything ties, and the number is still wrong.
The automatic check answered “did it balance?” The reviewer answered “is it right, and why not?”
What the reviewer adds
An expert review gives an AI team three things a pass or fail flag cannot:
- A score against a written rubric. We score completeness, accuracy and controls separately, so a team can see which kind of mistake its agent makes.
- A note that names the error. “FX difference booked at the invoice-date rate” is something a model can learn from. “Fail” is not.
- The corrected answer. The reviewer shows the entry as it should have been booked.
Why the reviewer’s credential matters
The right answer often depends on the rulebook. IFRS and US GAAP agree on the closing rate here, but they differ in other places, and so do tax codes and local practice. We match reviewers to the framework a task uses, and we check each reviewer’s license with the body that issued it before any paid work.
What a team receives
Each reviewed item ships as a JSONL record: the agent run, a score per rubric criterion, the reviewer’s note and the corrected answer. Every dataset comes with a rights record. The example in this post is illustrative.