TECH
DISPELLING MYTHS
ABOUT EXPLAINABLE AI
A Column by Caspar Chorus
Over the past five years, Artificial Intelligence has taken the world of finance by storm. As AI has steadily moved up the value chain, from analysis and monitoring, to decision support, to automated decision-making and more, financial institutions have come to depend on it like never before.
The recent advent of generative AI has opened vast new opportunities, and it is a matter of time before these Large Language Model-based applications will have enhanced or replaced the capabilities of scores of human finance professionals. The jury is still out on whether this development is on balance a good thing. But on one aspect there is a clear consensus in the world of finance as well as in other sectors: AI should be explainable, especially when its ‘decisions’ have a large impact on humans. This means that the general workings of the algorithms underpinning the AI should be fully understood and that any particular AI-output (e.g. flagging a potential client as potentially fraudulent) can be explained in terms of the input data. However, despite sounding like a laudable ambition, the idea of explainable AI is riddled with misconceptions and unfair comparisons.
"AI should be explainable, especially when its ‘decisions’ have a large impact on humans."
Let’s start with irrefutable facts that are well-established throughout the behavioral sciences: human intelligence (HI) is itself not ‘explainable’ at all. That is, we humans are notoriously bad at explaining the drivers and processes behind our own decision-making.
Fascinating studies have shown that, be it a choice for a particular brand of detergent or a vote for a particular presidential candidate, humans generally have a very hard time coming up with consistent explanations for their own behaviors. If we have this much trouble explaining our own behavior, is it fair to request that AI systems explain theirs? Related to this: the more sophisticated AI systems become, the more difficult it gets to explain their inner workings. This is exactly why we humans have such difficulties explaining our own actions; our brains are simply too complicated to be fully understood.
The exact same attribute that makes machine learning systems like Artificial Neural Networks so powerful is what makes them so hard to explain: they capitalize on the connections between mind-boggling – and ever growing – numbers of ‘neurons’. In other words, do not expect to ever fully understand such a system.
Let’s start with the fact, well-established throughout the behavioural sciences, that human intelligence (HI) is itself not ‘explainable’ at all.
Is that a problem? Not necessarily. Recent academic work finds that users of AI-powered products do not seem to care much about whether the AI is able to explain its workings or its decisions. When the product works, it works. Few people care to understand how it works or why it works the way it does. It follows from this that since the newest generation of AI (such as ChatGPT and its friends) delivers such strong performance, its users will likely care even less about understanding its inner workings.
I'm reminded of a related study that explored how civil servants selected algorithms. It was found that these expert users of algorithms traded off explainability against accuracy; that is, when choosing which algorithm to use, they were fine with accepting lower levels of explainability when the accuracy was high enough. When accuracy was low, an algorithm was selected only when it was perceived as explainable.
Perhaps even more subtle is the notion that explainable AI – if not properly designed – can actually lead to worse decision-making among experts. How does this work? A study found that when experts were confronted with advice by an AI-powered decision support tool whose inner workings they did not understand, they were reluctant to override it. However, when the AI provided an explanation behind its advice, experts felt more comfortable overriding the advice when they did not like it. Good news? Not in this case study, where it turned out that when the advice was not followed by the experts, this led to worse decisions. In other words: the AI knew better than the experts, and by explaining itself too well, it inadvertently gave them the confidence to ignore it.
To provide a more positive note: AI systems hold large potential to help us, the human experts, explain ourselves. What do I mean with that? Well, as elaborated earlier, humans are bad at understanding their own decision-making. AI systems, while easily discredited as being non-explainable, are oftentimes better at explaining their inner workings than humans are at explaining theirs. This is why a host of human biases, after ending up in AI-powered algorithms, are discovered – leading to uproar. In turn, the AI is tweaked to get rid of the biases and along the way, we, human experts, learn about our biased way of behaving. Isn’t that a much more positive frame for AI? To see it as a tool to explain and unbias ourselves.
How do we move from a world in which human experts are biased, AI systems are perceived as non-explainable, and people are generally unhappy with how the technology develops? Here, I have one very simple bit of advice: while many are proud of the EU’s ‘Brussels effect’ in regulating new technologies such as AI, I consider regulation a post-hoc fix that will not lead to a satisfactory societal outcome in the long run.
Rather than weathering the storm by cracking down on AI and its sometimes rogue developers, we all (including the financial sector) should work to create optimal atmospheric conditions. That means designing AI technologies in such a way that explainability is maximized, risk of biases is minimized, and humans and AI make the most of one another – a more stable long term forecast. At TU Delft’s Faculty of Industrial Design Engineering, this type of value-based design for AI is at the core of our mission, and I'm looking forward to discussing this topic further with the FMO community during the upcoming Future of Finance conference.
Caspar Chorus is Professor of Choice behavior modeling at TU Delft, Netherlands, Dean of the Faculty of Industrial Design Engineering, and co-founder of spin-off Councyl, which develops explainable decision support software.
