0.15%: how we got hallucinations low enough to let AI touch money

Let me tell you why this number keeps me up at night, and why getting it right is the whole company.

In most products, an AI hallucination is embarrassing. It invents a fact, you laugh, you move on. In banking, a hallucination does not hand you a wrong fun fact. It hands you a wrong account.

Picture it. A customer says "send my rent to my landlord." At a 2% hallucination rate, the AI can confidently produce a bank name that does not match, or an account name that belongs to a total stranger, and say it with the exact same calm certainty it uses when it is right. Two percent sounds small until you remember that two percent of thousands of transfers is real people's rent landing in the wrong hands. There is no "oops, let me correct that" after money has moved. In our world, a hallucination is not a bug report. It is a stranger holding your rent.

That is the standard we are actually being judged against. Not "is the chatbot clever." Is it safe to point at money.

We noticed it. So we shut the whole thing down.

At one point we saw it happening. Users on TwelveAI were experiencing hallucinations. Not catastrophically, but enough that I could see the shape of the risk, and I refused to let it grow.

So we did something most companies would never do to a live product: we shut the entire AI down. Not patched around the edges, not added a warning label. We took the intelligence offline and decided we were not turning it back on until we could trust it with a transfer the way you would trust a careful human teller.

I would rather have no AI than a confident wrong one. Foodr taught me not to break promises to customers. This was the same rule, applied to a much more dangerous surface.

We built a bench before we rebuilt the brain

You cannot fix what you cannot measure, and "vibes" is not a measurement when money is involved. So before we rebuilt anything, we built a bench, our own benchmark for hallucination, designed specifically for financial conversations: does the model ever state a balance, a bank, an account name, or an outcome that a tool did not actually return this turn?

That bench became our source of truth. Every version of the AI now has to earn its way past it before it is allowed anywhere near a real customer. No number on that bench, no launch.

Then we went and got smarter people involved

Rebuilding the brain was not something we did by guessing. We teamed up with a professor who put real research in front of us, work on how these systems reason and where they drift. We spent serious time refining that research, adapting it to the specific problem of financial grounding, and then we rebuilt our entire AI on top of it.

I am not going to give away exactly how we did it here, that part is our edge, and we earned it the hard way. What I will say is that the core principle is boring and absolute: the model may only tell a customer something a tool actually returned this turn. Everything else is architecture built to enforce that, relentlessly, measured against the bench, every single time.

The number

After that rebuild, we brought TwelveAI back with a hallucination rate of 0.15%.

To our knowledge, that figure is currently unmatched in the AI fintech space. I do not say that to boast. I say it because it is the number that decides whether AI belongs anywhere near a bank account, and we treat it as the most important metric in the company. Not user growth. Not revenue. This.

We got there because we were willing to turn our own product off, measure honestly, learn from people smarter than us, and rebuild. A lot of this happened late at night, after my kids were asleep, because it mattered that much to get right.

If you are going to let AI touch money, this is the bar. Not "impressive." Grounded. And you should be able to prove it with a number.

Want to see the discipline in action? Try the Playground, or read how we enforce grounding in the docs.

Ifeoluwa Alao Ojo

Founder & CEO, TwelveAI

Ifeoluwa is the founder and CEO of TwelveAI. Before this he founded Foodr, led mobile engineering at a microfinance bank, and ran the engineering team at Cantant as it scaled to 50,000 businesses. He is building TwelveAI so any bank or fintech can offer banking your customers can simply talk to.