Research area
Human-Centric AIThe need for explainable human-centric AI
AI systems perform tasks that normally would require human intelligence, and can accomplish them faster by rapidly consuming and analysing large amounts of complex data. That’s the theory. In practice, many instances of applied AI fail spectacularly. In some cases, we don’t even notice!
How does AI think?
AI is typically a black box. In many cases we don’t understand why AI systems make decisions or predictions. This enigma becomes more pronounced the more complex AI decision-making becomes. Without this understanding, we won’t know when AI gives us a wrong answer. To harness AI’s full potential requires human-AI collaboration, which combines the best of both worlds: the profound analysis and pattern-recognition capabilities of AI with empathetic human decision-making.
To accomplish this, NEC Laboratories Europe is developing explainable human-centric AI that will allow these systems to explain themselves to human users, empowering us to make better and more informed decisions.
The need for explainable AI
To deploy AI systems safely, especially in medium-risk and high-risk scenarios, we need explainable human-centric solutions that enable us to understand what a system learnt and why it decided or predicted what it did.
Harnessing human-AI collaboration
First, let’s examine what we mean by human-AI collaboration with an example of a biomedical researcher collaborating with an AI system (see Figure 2). The researcher wants to predict the side effects for a patient when one or more medical substances are taken at the same time.
The biomedical researcher’s first question is: “What happens if someone takes the drug paliperidone and the chemical element calcium at the same time?” A conventional AI system might provide a correct prediction, saying, “It will cause pain,” but would stop there. A noncollaborative AI system can’t answer the researcher’s next question, “Why do you think it will cause pain?”
A collaborative AI system would then explain that each substance activates the particular proteins, LPA and MMP. When the researcher asks about the relationship between those proteins, the AI system would also explain that LPA further increases the activation of MMP, which is already activated by calcium. Thus, paliperidone indirectly exacerbates the upregulation of MMP already caused by calcium, which leads to increased pain.
Collaborative AI systems can increase user productivity when they interact naturally with humans and explain their predictions in a way that helps users accomplish their tasks.
Benefits of human-AI collaboration
Human-AI collaboration enhances human decision-making and delivers three major benefits:
- Increased human efficiency: By performing low-effort tasks, AI frees up time for other human activities.
- Better decision-making: AI provides requested information quickly to support actionable insights.
- Improved processing capabilities: AI detects big data patterns impossible for humans to discern, leading to AI-assisted breakthrough discoveries.
Why are human-centric AI explanations necessary?
A typical AI setup that does not provide explanations works like this: Given some training data, a machine learning process produces a learned function. That function provides a decision or recommendation to a user and the interaction ends. Important user questions go unanswered, such as: “Why was this decision or prediction made?” and “How sure is the model about it?”
In contrast, given some training data, the explainable AI machine learning process returns a model that provides explanations along with its decisions and predictions.
When developers and users understand what the machine learning model learns, they can determine the value of its output, for example, with regards to bias. Developers can also audit explainable AI systems to assess the degree of legal compliance needed and understand why decisions and predictions were made. This helps foster trust amongst AI users and increases the adoption of AI for high-risk decision-making.
Currently, most research in explainable AI is algorithmic-centred: researchers explore how AI can be explained from a mathematical point of view. This approach does not consider what kind of explanations humans need – something that explainable human-centric AI accomplishes.
Upcoming regulations, such as the Artificial Intelligence Act
The need for explainable AI to ensure compliance, transparency and ethical use becomes all the more evident when we look at the pending Artificial Intelligence Act (AI Act) proposed by the European Union.1 Once in place, all AI systems deployed in the EU will be required to adhere to its regulatory and legal framework. Explainable AI’s ability to understand what AI systems have learnt will provide developers with a significant cost and time advantage in ensuring compliance. The draft EU AI Act currently specifies four risk categories of AI systems.
The lowest category is “Minimal Risk,” appropriate for spam filters, video games and in cases when the user needs to know that AI is being used and be able to opt out. “Limited Risk” refers to decisions or predictions made by AI systems that may pose some risk to users and therefore require transparency. AI chatbots that provide automated, low-risk customer service information is one example. “High Risk” includes domains that affect human rights or areas where AI will impact critical decision-making such as in areas of public services, law enforcement or medicine. Examples include large language models providing decisions or predictions for high-risk scenarios and autonomous vehicle systems.
For conformity assessments, explainable AI techniques can be applied to tell us what the AI system has learnt. After deployment, continuous market surveillance is called for. The highest risk category defines situations of unacceptable risk, such as threats to safety, livelihoods or human rights. In these areas, the use of AI will not be allowed.2
The AI Act also mandates that AI systems cede control to developers and users when desired or needed, which will make human-centric AI a key requirement for most AI systems moving forward. This is especially relevant when considering the rapid advancement in AI. While the EU is at the forefront of pending AI regulation, non-EU countries are following suit. In the United States, Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence was issued by President Biden on October 30, 2023.
Important concepts in explainable AI
To help us choose the most suitable explainable AI technology for a particular use case we need to understand the key explainable AI concepts.
Transparent versus opaque AI models
AI models use two types of decision-making paradigms: transparent and opaque. A transparent model lets us follow how the model arrived at a decision, for example, a decision tree, where-as opaque AI models gives us black-box AI.
The figure below shows two decision trees used to help us decide if someone will be granted credit. The first tree has only two rules and is easily understood. We see that a person receives credit if they fulfil two conditions: 1. earn more than fifty thousand Euro a year and 2. have not defaulted on credit.
The second tree is more complex and represents the type of information required to optimize most AI decision-making tasks. It consists of many hierarchical rules that have different cut-off points. First, the system checks if someone earns more than amount “a,” checks another condition, then returns to refine the amount the person earns by comparing it to amount “c” and “d.” After even a few steps, a decision tree can become too complex for a human to follow.
With opaque AI models, humans can’t follow how a decision is made (even with smaller models). This is often called black box AI. We observe this with neural network-based models which, although effective, are hard to understand; it is also difficult for developers to verify that they work as intended.
As a result, users may not trust their decisions or predictions. To resolve this, researchers are developing explainable methods for opaque models that improve their technical components. However, if human-AI collaboration requirements are ignored then humans may still not understand these methods.
Post hoc versus built-in explanation methods
When using an explainable method for an opaque model, we need to decide whether a post hoc or built-in explanation method should be used. Post hoc methods are added after a model has been trained and can explain the decision-making of any AI system. Well-known examples include Lime and Shap.3,4 Built-in explanation methods are included during the models development; Examples include influence functions and prototype-based learning.5,6
Evaluating faithfulness and plausibility
To choose the right method we must also consider the model’s use case. To help with this, we use the definitions of faithfulness and plausibility, which are important concepts when considering the success of an explanation method. When designing AI systems, faithfulness and plausibility account for the trade-off needed between how complex or plausible an explanation is and how well we understand the model’s reasoning process (how faithful it is). For high-risk scenarios, a method should have a high degree of both.
Faithfulness indicates how accurate an AI system’s explanation of its reasoning process is. If someone asks, “Why is the sky blue?” the system may answer, “Because someone painted it blue.” While a valid explanation, it is neither true nor faithful. Explanation methods will vary in how faithful they are. A high amount of faithfulness is needed when the model’s decision influences a high-risk scenario.
Plausibility assesses how clear the explanation is to a human, in other words, how human-centric it is. This is important because an explanation achieves its goal only if it helps a human user. As with faithfulness, a decision or prediction can vary depending on how plausible it is.
It is difficult for an explanation method to be both highly faithful and strongly plausible. A trade-off between the two is usually needed.
An on-board vehicle navigation system detects oncoming heavy traffic and recommends an alternative route to the driver. Based on the driver's local road knowledge and radio traffic reports they may consider the recommendation plausible. They would not consider driving into a lake – this recommendation would be implausible.
Decision trees are often highly faithful but vary in how plausible they are. Smaller trees are more plausible than larger ones. However, how plausible a decision tree is will depend on the extent to which its attributes make sense to humans. Even a small decision tree can lack plausibility if its attributes are not understood, which highlights the importance of providing human-centric explanations.
Post hoc methods, like those proposed by Lime, sometimes have a very low degree of faithfulness. We should avoid trusting explanations of a model’s decision or prediction that uses those methods. For example, consider a person who was denied a loan, with the explanation that their income was not over the minimum income threshold for loan approval. After a salary increase to exceed the minimum threshold, the decision doesn’t change; they are still denied a loan. This shows that the explanation was neither true nor faithful.
The goal of explainable human-centric AI is to provide the right trade-off between faithfulness and plausibility. For example, a high-risk task requires the decision attributes to be highly faithful, which may lead to a drop in how plausible the AI system’s decision is, meaning it will take a human expert longer to understand the explanation. However, because of the high-risk nature of the task this may be acceptable.
An AI system recommends a clinical treatment for a patient. The decision attributes for the recommendation can be considered faithful because of the system’s complex analysis of the patient’s clinical data paired with possible treatment recommendations.
While the system’s recommendation may be suitable it has yet to be verified as plausible and needs to be validated by a doctor. This may take an extended amount of time.
In other situations, when the user has little time, and the task is low-risk, a less faithful but highly plausible method may be suitable, for example, a movie recommendation. An overview of how different explanation methods compare when considering faithfulness and plausibility can be seen in the figure below.
Post hoc and built-in explanations: The right explanation method depends on the use case
Building on our understanding of what it means to have faithful and plausible explanations for AI models, let’s examine how these explanations apply to post hoc and built-in explanation methods.
Post hoc methods have lower faithfulness than built-in methods, and their plausibility can only be slightly influenced. Built-in methods are designed to restrict an AI’s reasoning. The AI developer determines how plausible these methods are. However, using built-in methods often requires replacing the AI system with a new one. When this can’t be done using built-in methods are not possible.
Technology for human-centric explanations
Gradient Rollback
Gradient Rollback is a highly faithful AI method that modifies how neural networks are trained. Developed by NEC Laboratories Europe, the method explains the prediction of a neural network by highlighting which training examples made the prediction. Compared to other explanation methods, Gradient Rollback offers more context and provides a user interface allowing users to ask for additional information.
Gradient Rollback is suited for training new neural networks where higher than average faithfulness is needed, for example, to explain a recommendation. It is ideally suited for AI systems used by operational decision-makers to make multi-criteria decisions and provides essential context for informed decisions.
Prototype-based learning
Prototype-based learning can either be used as a transparent model or explain opaque models and seeks to accomplish two objectives:
- Produce an explanation method that provides a clear explanation of its reasoning process, making it suitable for high-risk areas
- Produce a method that can explain the decisions or predictions of an existing system
NEC Laboratories Europe is extending the scope of prototype-based learning by incorporating human-centric AI concepts to ensure that explanations from prototype-based models are more plausible while achieving the highest amount of faithfulness. With that accomplished, prototype-based learning will be able to provide different types of explanations, such as training examples and counterfactuals. Once mature, applications of human-centric AI can be further extended by designing more customized concepts.
Let’s look at how prototype-based learning can be applied to image object classification by first creating an abstract representation of a car. We then identify and classify an image as a car by measuring its closeness to the initial abstract representation. High faithfulness-by-design is achieved by mapping new instances of the car to the closest vehicle prototype, which then can serve as an explanation. We make the approach more human-centric and plausible by ensuring that the learnt prototype represents a human-understandable concept.
This technology can be used for any AI domain and, in contrast to many other methods, can be applied in high-risk areas, for example, by supporting doctors during diagnosis by providing both faithful and plausible explanations.
Prototype-based learning is designed to be explainable, and its explanations can be trusted. Experts can inspect whether a decision or prediction can be trusted based on the explanations given. Because of the different components that are included in machine learning, different explanations can be provided, resulting in a human-centric solution that is highly plausible.
Conclusion
The rapid acceptance of complex AI for general use, like large language models (LLMs), marks a milestone in its adoption. LLMs have had great success when applied to low-risk tasks but are not yet suited to medium-risk and high-risk scenarios. In the short time that complex AI has been in the public domain, it has been unintentionally and deliberately misused, and the consequent risks are increasing. Human-centric AI will have a great affect in helping guide its rapid development and responsible use.
Authors
NEC Laboratories Europe Group Research Manager, Human-Centric AI, and Chief Research Scientist
References
- AI Act, European Commission, https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai,accessed November 24, 2023.
- Ibid.
- Marco Tulio Ribeiro, SameerSingh, and Carlos Guestrin, “‘Why should I Trust You?’ Explaining the Predictionsof Any Classifier,” Proceedings of the 22nd ACM SIGKDD International Conferenceon Knowledge Discovery and Data Mining, 2016.
- Scott M. Lundberg and Su-InLee, “A Unified Approach to Interpreting Model Predictions,” Advances in NeuralInformation Processing Systems 30, 2017.
- Pang Wei Koh and Percy Liang,“Understanding Black-box Predictions via Influence Functions,” International Conferenceon Machine Learning, 2017.
- Atsushi Sato and Keiji Yamada,“Generalized Learning Vector Quantization,” Advances in Neural Information ProcessingSystems 8, 1995.
Figures
Example of human-centered AI highlighting the side effects of drug interaction
B. Malone, A. García-Durán, and M. Niepert, “Knowledge Graph Completionto Predict Polypharmacy Side Effects.” Data Integration in the Life Sciences,S. Auer, ME Vidal (eds). DILS 2018. Lecture Notes in Computer Science, vol11371. Springer, Cham. https://doi.org/10.1007/978-3-030-06016-9_14.
A comparison between AI system methods and components when considering faithfulness and plausibility
7. Marco Tulio Ribeiro, SameerSingh, and Carlos Guestrin, “‘Why Should I Trust You?’ Explaining the Predictionsof Any Classifier,” Proceedings of the 22nd ACM SIGKDD InternationalConference on Knowledge Discovery and Data Mining, 2016.
8. Scott Lundberg and Su-In Lee,“A Unified Approach to Interpreting Model Predictions,” Advances in Neural InformationProcessing Systems 30, 2017.
9. Ashish Vaswani et al., “AttentionIs All You Need,” NIPS 2017.
10. Mukund Sundararajan, AnkurTaly, and Qiqi Yan, "Axiomatic Attribution for Deep Networks,” Proceedings of the 34th International Conference on MachineLearning, PMLR 70:3319-3328, 2017.
11. Atsushi Sato and Keiji Yamada,"Generalized Learning Vector Quantization,” Advances in Neural InformationProcessing Systems 8, 1995.
12. Carolin Lawrence, TimoSztyler, and Mathias Niepert, “Explaining Neural Matrix Factorization with GradientRollback,” 35th AAAI Conference on Artificial Intelligence, 2021,https://aaai.org/Conferences/AAAI-21/.