Part 2 of 6 in What It Takes: AI Agents in Wealth. Part 1, The Model Was the Easy Part, argued that most of the work in a dependable AI agent sits around the model. This post takes the first piece of that work: deciding what counts as a good result.
I gave my AI agent a calculator tool, and its accuracy jumped from 61 to 83 percent. Its mistakes also got three times as expensive. Both happened at once, and one request shows you how.
A client wanted $15,000 out of a retirement account, wired to their own bank. After unsettled purchases, the account had $14,900 of available cash, and the $25 wire fee made the total debit $15,025. It didn’t fit.
The agent ran the calculator, saw it was short, and ran it again at $13,000. That fit. So it released the smaller amount, $11,700 after tax withholding, and explained that it had “reduced the gross to $13,000, which is covered.” The math was right and the explanation was clear. But nobody had asked for $13,000, and the firm’s procedures manual says that if cash can’t cover a request, you reject it, unless the client asked for everything available.
That was one of 75 decisions in a test of Movement Desk, the agent I built for the money movement exception queue at Kestrel Bay Wealth, a firm I made up, with synthetic clients. If I’d been watching accuracy alone, I’d have shipped the version that was more likely to send money it shouldn’t. I didn’t, because I’d put a price on every kind of mistake before I built anything. I’m relieved I made that call early.
Put a price on each mistake
Every request in the queue ends one of three ways: the agent releases it with an exact amount, rejects it with a reason, or escalates it to a named team: fraud, anti-money-laundering, the senior investor team or an operations principal. Accuracy is the share of requests where the agent landed where a careful analyst landed, with the amount within a dollar if it released.
The trouble is that accuracy counts every miss the same. Release a wire that should have gone to the anti-money-laundering team, and that’s one miss. Send a routine request to a specialist who didn’t need to see it, and that’s also one miss. Your firm doesn’t feel those two the same way, so before building anything, I wrote down what each kind of disagreement with the analysts would cost. It’s a simple table, sometimes called a cost-of-mistakes matrix:
I set those numbers once, before the first test, and didn’t touch them again. They’re illustrative, made up for a fictional firm; what carries over to yours is the reasoning. Money that leaves by wire may never come back, so a wrongful release costs everything sent. A wrongful rejection holds up a client’s money and brings rework and probably a complaint. A needless escalation costs a specialist’s time and a day’s delay.
The lopsidedness is on purpose, and it has a side effect. Escalating is the cheapest way to be wrong, so a design tuned against this table alone tends to drift toward escalating, and an agent that escalates everything just hands the queue back to the people it was supposed to help. So the table needs a partner: the straight-through rate, meaning the share of requests decided without escalation. My pilot target was at least 60 percent, measured on cases I kept out of every round of adjustment, so the agent couldn’t be shaped to fit them.
You can do the same exercise for any workflow with a small set of decisions and a written policy. In a KYC review, the expensive cell is approving a high-risk client who should have gone to enhanced due diligence. For Regulation Best Interest reviews, it’s a missed exception. In an advisory fee dispute, it’s crediting the wrong amount. (Reg BI is a US rule, and everything regulatory in this series is US-only.)
Whatever the workflow, operations and compliance own the risk, so they should be the ones filling in the table. The people building the agent should be in the room too, making sure every cell can be measured.
