Not every AI question needs a chatbot

Working out which client an ambiguous email belongs to is a decision, not an essay. We have put a new kind of model behind it — one that returns a typed answer and a number saying how sure it is — and it is running alongside the old one until the evidence says switch.

Every email that lands in a connected mailbox has to end up against a client. Most of that is not a judgement at all: the address matches a contact, and AccountingFlow files it. The interesting ones are the rest — the outsourced finance manager who looks after three of your clients and writes about all of them from the same address, the group accountant sending for whichever subsidiary is on their mind that morning.

For those we go to a second stage: a shortlist of the clients that sender could plausibly mean, and a model asked to pick one. We only bother when somebody at the firm has already corrected that exact sender before, so there is real history to reason from.

What we were doing, and why we stopped

Until now that second stage asked a chat model. It is the obvious thing to reach for, and it works: describe the email, list the candidates, ask for the answer as JSON.

Then comes the unglamorous part. Strip the code fences the model wrapped its answer in. Parse it. Check that the client it named was actually one of the ones we offered, because nothing stopped it inventing one. And at the end of all that you have an answer with no idea how sure the model was — the same flat sentence whether the email named the client in the subject line or could genuinely have been either of two.

That is a lot of scaffolding around what is, in the end, a choice between two rows in a database.

A model that does not write

So we have wired a second engine in behind that one decision. Jev, from TypeSafe, is what they call a System One model, and it generates no text at all. You hand it the state — here, the email and the candidate clients — plus a set of typed questions, and it answers them all in one parallel pass.

Three things follow from that, and all three matter here.

• It can only return an answer we supplied. A choice question is answered with one of the options you gave it. Naming a client that was not on the shortlist is not something it declines to do; it is not something it can do. The defensive code goes away.
• It says how sure it is. Every answer comes back with a calibrated confidence between 0 and 1, and the full spread across the options — 91% to one client, 9% to the other. That is the number we had been missing.
• It is quick and it is cheap. Answers land in a fraction of a second, and there is no output to pay for, because there is no output. Importing email cannot afford to wait around, and this does not make it.

The number we actually wanted

Confidence is the part that changes the behaviour, not the price.

A chat model asked to pick one of two clients will pick one of two clients. It has no way to shrug. With a confidence figure, our code does: below a floor we set, the answer is thrown away and the email goes where it went before — into "Check assignment", for a person to decide.

And because a choice has to choose something, we ask a second question in the same breath — is this email clearly about exactly one of these clients at all? — and require both to clear their floor before anything is filed. Two cheap questions, one round trip, and a tie-break that knows when to keep quiet.

An automated guess that is right 70% of the time is not seven tenths of a good feature. It is a filing error in one email out of three, found later by whoever goes looking for it.

We did not just switch it on

New engine, real client mailboxes: the interesting question is how you would know it was better, and we would rather answer that with data than with a launch.

So the pilot runs in shadow. Both engines are asked. The chat model's answer is still the one used. Every disagreement is logged — which sender, what each engine said, how sure the new one was — and joined up to what the person at the firm went on to correct, which AccountingFlow already records. That tells us not only whether Jev is better, but where its confidence floor should sit: in our data, on our customers' post, rather than on a benchmark.

The fences around the decision do not move either way:

• Both engines choose from the shortlist the deterministic matcher produced. Neither can reach a client that was not already a candidate.
• A tie-break made this way is still recorded as a low-confidence assignment, still labelled as made by AI, and still lands in the review queue for a human.
• If the new engine is slow, unavailable or unsure, the old path runs exactly as it did before. If both are unsure, the email waits for a person — which is what it did before any of this.

Staying on top of a field that moves monthly

The model landscape changes faster than any accounting practice should have to care about. Something cheaper, quicker or better shaped for the job turns up every few weeks, and most of it will not matter.

What you want is for that to be somebody else's problem — ours — and for it to be a small one when it arrives. Because this decision already lived behind a single seam in the code, putting an entirely new kind of model behind it was a flag and a couple of hundred lines, not a rewrite. We can try it, measure it on real work, and turn it off again with one setting if the numbers disappoint.

There are other decisions in AccountingFlow shaped the same way — is this receipt from HMRC, is this email relevant to the client at all, which status does this search mean — and every one of them is currently a chat model writing words we immediately turn back into a decision. This is the first one we have changed. It is deliberately the smallest.