Explainability Is a Liability Shield

Most boards treat explainability as a technical feature, something the data science team should tick off before a model ships. In a regulated institution, that is a category error. Explainability is not a feature. It is a liability shield, and most institutions are carrying the risk uninsured.

Here is the distinction that matters. A model can be accurate and still be indefensible. Accuracy tells you the model is usually right. Explainability tells you why it decided what it decided in the one case that is about to be challenged. Those are different assets, and only one of them is any use in the room where the challenge happens.

The bill arrives when a decision is contested

An automated decision creates value quietly and creates liability all at once. A credit application is declined. A transaction is flagged and a relationship is frozen. A customer is moved into a higher risk tier. For thousands of cases, nothing happens. Then one case is contested, by a customer, a regulator, an ombudsman, or your own internal audit, and the institution is asked a single question: why did you decide this.

At that moment you hold exactly one of two positions. Either you can reconstruct the decision in terms a non specialist will accept, or you cannot. If you can, the model was a controlled instrument. If you cannot, the model was an exposure that had not yet been called. The accuracy of the other ten thousand cases is irrelevant to the one on the table.

The regulator is not testing whether it is accurate

This is the part that surprises technical teams. When a supervisor examines an automated decision, accuracy is rarely the question. The question is whether the institution understood, controlled, and can account for what its own system did.

In every regulated system I have put into production, the question that came back was never how accurate is it. It was: show me why it did this.

Model risk management has expected this in banking for years: a model you cannot validate or interrogate is a model you are not allowed to lean on, however well it scores. The EU AI Act pushes the same logic across a wider set of high stakes uses, including credit and insurance, with obligations for transparency and human oversight. And where a decision materially affects an individual, data protection law gives that person standing to ask how it was reached. None of these regimes are satisfied by a high accuracy number. They are satisfied by an institution that can explain itself.

Accuracy without explainability is an uninsured position

Read it the way you would read any other exposure on the book. An accurate but opaque model is a position that pays a steady return and carries a tail risk you have not provisioned for. It works, it works, it works, and then a single contested decision arrives with a regulator attached, and there is nothing set aside to cover it.

Explainability is the provision against that tail. It is not there to make the model better. It is there so that when the decision is called, the institution settles it from reserves rather than from its reputation. Framed that way, explainability stops being a cost the data science team is asked to justify and becomes a control the board already knows how to value.

Post hoc explanation is not the same as an explainable decision

A warning, because this is where institutions buy themselves false comfort. Bolting an explanation tool onto an opaque model produces an approximation of why the model might have decided something. That is useful for debugging. It is not the same as a decision you can defend, because the explanation and the decision are two different objects, and a sharp challenger will separate them in minutes.

If the explanation is generated after the fact by a second system trying to guess the first, you have not closed the exposure. You have documented it. The decisions that actually need a shield are the ones where you can trace the reason through the system itself, not the ones where you can assemble a plausible story about it afterward.

The test you can run this quarter

Take your single highest stakes automated decision. Not the pilot, the one already in production, affecting real customers or real capital. Ask two questions in the room, out loud.

If a regulator challenged one specific output from this system tomorrow, what exactly would we hand them. And would the person who owns this decision be comfortable defending that answer under examination, or would they reach for the phrase the model determined it.

If the honest answer is that you would hand over an accuracy metric and a model card, you are describing the position, not the shield. The distance between those two is the liability you are carrying unprovisioned.

The position I hold

Explainability is not the ethical garnish on an AI program. In a regulated institution it is a financial instrument: the provision that lets you absorb a contested decision without absorbing the reputational loss behind it. The institutions that treat it that way will price it, own it, and fund it like any other control. The ones that treat it as a technical nicety will keep booking the return and discovering, one contested case at a time, that they never set aside the reserve.

Accuracy tells you the model is usually right. Explainability is what you reach for on the day that is not enough.

Where in your book is that provision missing today, and who would notice first if it were called.