A customer learns that his tariff is being switched to PrePaid. The decision was made by an AI. He asks: Why?
There are two aspects to this question. The business aspect: Do I myself understand what the system has done? The legal aspect: Do I need to be able to explain the process if a customer disputes it or a court asks?
The second question is not the subject of this article. It is a matter for the legal department. What matters here is this: if the answer is ‘yes, I must’, this has implications for the AI involved in the process. That is precisely where the choice of technology becomes crucial.
What explainability is
Explainability is the ability of an automated system to make its decision transparent. This applies to rule-based systems just as much as it does to AI. Anyone who cuts off a customer’s electricity supply after five reminders is obliged to explain why there were five reminders, why the supply was cut off, and why no assessment was made as to whether the customer was facing hardship. This requirement is not new and is not tied to any particular technology.
An example illustrates this clearly. ‘These customers are likely to cancel their contracts in the next quarter‘ is a statement. ‘The reason is that waiting times for customer service have increased in recent months and usage has fallen‘ is an explanation. A system must be able to provide this explanation when required.
With a rule, this isn’t a problem. With AI, it depends on the model used. Some can do it; others cannot.
Where explainability becomes key
There are areas of application in which explainability is a requirement. The list is familiar from the energy sector: creditworthiness assessments when concluding contracts, decisions to suspend supply under supply contracts, adjustments to terms and conditions without the customer’s consent, fully automated terminations, and profiling within the meaning of Article 22 of the GDPR. In 2023, the European Court of Justice (ECJ) expanded the scope of application in the Schufa ruling. Even a scoring value can be regarded as an automated decision.
A rough distinction is helpful in everyday life. Where a decision is based on clear facts and a well-drafted clause in the terms and conditions, ‘if-then’ logic is often sufficient. Example: five unsuccessful reminders lead to the account being suspended. In this case, there is no need to incorporate an AI model. Where, on the other hand, a decision is based on profiling, i.e. the assessment of the customer based on a multitude of characteristics, AI comes into play. Example: a prepaid tariff based on an external score combined with customer data in the operational system. In this second case, explainability becomes mandatory.
One area of application extends beyond the municipal utilities sector, but is instructive: AI in recruitment. The AI Act explicitly identifies this in Annex III as a high-risk application. Nevertheless, it is used on a daily basis in many companies, on the grounds that ‘a human makes the final decision’. Anyone relying on this argument should be aware of the ECJ’s Schufa ruling. Anyone who relies heavily on automated pre-selection has made an automated decision.
This article does not assess whether an obligation applies in a specific case. That is a matter for legal evaluation. What matters here is that, as soon as the requirement is on the table, it has implications for the choice of technology.
Which AI can provide explainability?
The answer depends on the model class.
Analytical AI (machine learning). Models such as decision trees, linear regression, random forests and gradient boosting learn from historical data and generate forecasts. It is possible to trace which input variables contributed to the result and in what way. The logic is either directly visible or can be deduced retrospectively. With more complex models, the explanation is not perfect, but it is possible. In most use cases, this is sufficient.
Generative AI (language models). Models such as ChatGPT, Claude or Gemini are structurally different. They have billions of parameters and generate responses based on probability distributions of words (see Dice or Clockwork). The models cannot substantiate their decisions in the traditional sense. When ChatGPT formulates a justification, this is itself a probability calculation, not a retrospective account of the decision-making process.
What this means in practice. An application with a requirement for explainability can be built using analytical AI. This is not possible with a language model. Anyone who uses generative AI in such a process is building a system that, by its very structure, cannot meet this requirement.
Typical applications, different tooling choices
Credit rating. Explainability is key here. A language model cannot be used as a credit assessor; it cannot justify a score that would stand up to challenge. Analytical AI with transparent decision-making logic is the appropriate solution here. Churn prediction. Here, explainability is essential for business, even if there is no legal obligation in the strict sense. Knowing why a customer is leaving allows targeted measures to be taken. A list of names without any justification is of little use to the sales team. Here, too, analytical AI is the right tool.
Handling enquiries and internal research. Here, explainability is not essential, because a human retains control or errors can be easily corrected. For individual operations, a language model is suitable, for example for translating tariff terms into plain language or classifying the topics of incoming enquiries. As part of an agent, it can operate in multiple stages. The research agent searches internal documents and provides the answer with sources. The response agent drafts a reply based on contract and usage data, which the case handler then reviews. The language model remains a tool. The human makes the decision.
What to do when selecting tools
Check the legal framework. Clarify with the legal department whether explainability is required for this process.
Make the architecture transparent. Ask the provider or the IT department which model is essentially making the decision. Analytical AI or language model?
If explainability is mandatory and a language model is making the decision, the architecture is unsuitable. “Explainable output” from a language model remains a marketing ploy.
To carry out a structured suitability assessment, I have developed the HOIKEI methodology. It combines the assessment of process suitability with the question of the correct model class.
Anyone who understands the process and selects the appropriate model class has already won the battle.