Towards Safer Large Language Models

The astonishing performance of large language models (LLMs) for many tasks has led to an exponential increase in real-world applications. They are reshaping the way society and industry interact with computer systems and textual information, as well as expanding our creativity. These characteristics, along with their flexibility in processing and interpreting natural language, let them accomplish a diverse range of tasks, such as generating textual content, powering virtual assistants and providing insights from large-scale data analysis.

However, as we explore the many benefits that LLMs offer, we must also tread cautiously, acknowledging their limitations and considering possible mitigation strategies. We need to be able to identify suitable LLM use cases and deploy strategies to reduce risk, such as prompting an LLM to explain its reasoning path in natural language and designing systems that keep people in control. Managed carefully, we can benefit from the great potential of LLMs while minimizing their risks.

LLMs are a family of neural network models for text processing generally based on networks that implement the Transformer architecture.¹ Unlike past language models trained on task-specific, labeled datasets, LLMs are trained using unsupervised learning on massive amounts of data.

The training objective of an LLM is to predict correct output based on an input prompt. The simplicity of the objective and the ability to learn from unlabeled data means these models can be scaled to ingest massive amounts of data. This training regime unlocks the model’s ability to solve previously unknown tasks and acquire emergent behaviors.^2,3 For instance, LLMs can answer questions, generate stories, extract information and summarize text. They can also reason, solve problems, generate creative text and be adapted to different domains.⁴

LLMs show significant capabilities in many tasks, often outperforming more narrowly focused natural language processing (NLP) or machine learning models. LLMs can also be used as-is for target tasks with no need for modification, further training or domain-specific datasets, which is unleashing an unprecedented pace of development and innovation in machine learning applications. LLM capabilities include:

Responding fluently – LLMs demonstrate great versatility in generating fluent text in different styles and contexts and are able to switch from colloquial prose to poetry to formal domain-specific writing, such as text for legal documents. Combined with a seamless ability to handle text in multiple languages, this capability of LLMs unlocks a universal text interface for any system’s input and output.

Inferring tasks – Following instructions, LLMs can often perform a task without referring to examples, sometimes referred to as zero-shot learning. If more instruction is required to teach an LLM how to perform a task, examples can be provided in the LLM text prompt field and the LLM can extract the target task from this. This capability is called few-shot learning and is particularly useful when a verbal description of the task might be ambiguous or otherwise difficult for the user.

Producing code – Because LLMs are able to process programming languages as well as natural languages, they demonstrate other interesting capabilities: generating working programs and programming code, understanding and explaining source code, fixing bugs in code snippets and translating across programming languages and software libraries.

While LLMs have great potential, they also have significant limitations. For instance, LLMs might generate incorrect output, a phenomenon sometimes called hallucinations.^5,6 In these cases, answers are imprecise, incorrect or even completely made up. 

Hallucinations are particularly dangerous when complex answers mix facts and false information. The user might also trust the response, falling victim to the argument from authority fallacy. Other key limitations include:

Lack of common sense – Although they excel at generating human-like text, LLMs often lack common-sense understanding. Given an input, they are trained to predict the next words based on statistical patterns in their training data. This can lead to factual inaccuracies and illogical responses. For this reason, LLMs have also been called stochastic parrots.⁷ Complex reasoning tasks where LLMs might fail include topics such as multistep, arithmetic, social, temporal or multimodal reasoning (due to a lack of understanding of the physical world).⁸

Biased training data – Prone to inheriting biases from their training data, LLMs may perpetuate or even amplify societal biases and stereotypes. This can affect the way LLMs generate text and make decisions. For example, many LLMs are predominantly trained on English data and, therefore, likely to produce outputs conforming to the culture of English-speaking countries.

Similarly, if an LLM is trained on social media data, it may exhibit bias or discriminatory views present in that training data. Addressing LLM bias is a significant challenge.⁹ For example, it requires careful curation of training data and ongoing monitoring to mitigate unintended consequences.

Harmful information – One under investigated issue is the hidden and harmful capabilities LLMs might have. For a start, it’s not fully known how safe the documents are on which these models have been trained. While LLMs often refuse to reveal what harmful information they know when prompted, adversarial prompting has been shown to succeed in causing them to reveal harmful information, such as instructions about how to steal someone’s identity or build an explosive device.¹⁰

Given the advantages and limitations of current LLMs, two key questions are, “What are good application areas?” and “What can we do to facilitate the safer use of LLMs?” To answer these questions, several approaches are available:

Classifying risk – Choosing the right strategy to reduce the risk of using LLMs inappropriately should begin with a use-case risk assessment. Ask yourself the question, “How much risk is there if I use this LLM for that task?” For example, there are no serious consequences if you use an LLM to recommend a book that doesn’t exist. In contrast, using an LLM to generate a medical report for a patient is high risk. If the medical report contains a hallucination, the doctor might choose the wrong treatment or even a dangerous one.

To assess risk, we can employ the risk definition outlined by the European Union’s Artificial Intelligence Act¹¹ (see Figure 1). The Act identifies four risk categories: Minimal Risk – users must be informed about AI usage and have the option to opt out; Limited Risk – transparency is required; High Risk – a conformity assessment should be run before AI can be deployed within the EU; Unacceptable Risk – AI may not be used.

Assessing accuracy – While LLMs generate text, the information they produce may be incorrect, making them useful in applications where strict correctness is not an issue, for example, when writing fiction. They can also be of value when correctness is important, if their responses are considered carefully by humans.

It is not uncommon for mistakes to occur during tasks being carried out by human experts. To avoid these, intermediate and time-consuming human reviews are often carried out. To reduce the time needed for these, LLM-based machine learning applications can undertake initial text generation tasks and seamlessly coordinate these with human experts as part of task workflows.

Identifying errors – One way to reduce the risk of overlooking incorrect LLM output is to explicitly ask the LLM to provide a natural language explanation behind its reasoning. For example, if we ask an LLM to solve a text-based math problem with multiple steps, it might become confused and provide an incorrect answer. As humans, we have no way of ascertaining why the LLM provided its answer without running the calculation ourselves, which defeats the purpose of asking the LLM!

This type of prompting aids the LLM in producing the correct answer as well as providing the reasoning path that led to it. However, there is one potential drawback with this approach: The engineering technique used to prompt the model is currently very brittle; adding a certain phrase cannot guarantee a correct answer and changing the phrase slightly could revert to an incorrect result.

Implementing controls – As it is impossible to ensure that LLMs work 100% correctly, what can be done to increase their safe use? We can facilitate this by designing applications that include human-centric LLMs. This increases the control humans have when using these applications.¹²

Typically, the amount of automation a computer application provides is seen on a single axis ranging from low to high. But this can be extended with an additional axis that embodies the amount of control we give to a user¹³ (see Figure 2).

Applying this paradigm to LLMs¹⁴ leads to three possible scenarios. First, without LLMs, humans are in full control of writing texts. Second, when LLMs generate text that is blindly trusted by the human user, there is a loss of control and the user is exposed to the dangers of LLMs. We can mitigate this with the third scenario: offering humans the right tools to restore control through innovative technologies.

To help ensure that correct LLM output is provided, NEC Laboratories Europe has created the Safe LLM Framework that consists of two technologies – LLM Explainer and Fact Checker. Developed by NEC Laboratories Europe, these machine learning models verify LLM output and enable the safer use of LLMs, minimizing the risk of hallucinations, falsehoods, biases and other problems that can affect output quality.

LLMs are prone to hallucinations, which makes them difficult to use in high-risk domains like medicine. Consider, for example, a summary of a medical instruction, where the LLM produces the sentence, “The patient should take a 50 mg pill of Drug X three times a day.”

However, when consulting the original document, we discover that this is a hallucination because the original instructions recommend the patient take a 5 mg pill of Drug X once a day. The LLM’s summary is a hallucination, recommending the patient take a 30-times higher dosage. If the patient blindly trusts the LLM and follows the instruction in the LLM’s generated summary, it could lead to serious consequences for the patient, perhaps even a lethal overdose.

To avoid such problems, LLM Explainer by NEC creates links between the generated LLM text and the original input (where information that LLM Explainer links to is already part of the input query to the LLM). This allows users to verify that the generated information from the LLM is correct. Similarly, we can highlight information present in the input but missing in the summary.

LLM Explainer can also be used for other tasks where we have access to the original text. For example, in the case of questions and answers, the user can provide a question on which the answer should be based. With LLM Explainer, we can map the generated answer back to the input text that the LLM used to construct its answer. Overall, LLM Explainer is a tool that enables the user to understand LLM output phrases that are generated by their input phrases.

In contrast, in some use cases, the relevant information is not provided alongside the input query, making LLM Explainer not directly applicable. Instead, we can use Fact Checker by NEC to verify LLM-generated information by comparing that information to trustworthy sources. This can alert the user to potential hallucinations as well as identify fake news.

A traditional LLM fact checker is an AI system that has been specifically trained for a task and adheres to a fact-checking pipeline that can be described as follows:

1. A given text is broken down into phrases.
2. The fact-checker identifies phrases that require fact-checking.
3. If the fact-checker determines a phrase should be checked, it becomes a claim.
4. The fact-checker retrieves relevant reference documents related to the claim.
5. Finally, using the reference documents, the fact checker determines whether the claim is true or false.

However, standard fact-checkers cannot explain why a certain claim is classified as true or false. A human would still have to run a verification, significantly reducing the benefit of the fact-checker. Using Fact Checker by NEC, we can increase the usability of an LLM by ensuring the fact-checker also outputs its reasoning path (see Figure 4). This is a series of sentences that combine known and trusted facts to ascertain whether a claim is factually correct or not.

Supplying a reasoning path is an important fact-checking capability provided by NEC’s Fact Checker. NEC is further improving this by enabling LLMs to explain the reasoning of their output (building on the supplied reasoning path) and improving fact-checking benchmarking for LLM sources of truth (the original, factually correct documents from which LLMs derive their answers.) 

Current LLM-based AI systems provide evidence documents that are used to determine if claims are true or false. However, they do not explain why or in what way evidence refutes or supports a claim. Adding explanations will further enhance the usability of these systems by speeding up the document verification process. 

Many existing fact-checking benchmarks for LLM sources of truth have limited capabilities and are designed for use only in specific industries or domains. For example, many benchmarks assume the correct evidence document is already given, whereas, in reality, it may have to be found first. To fact-check a claim, an internet or specific database search will be required before any claim can be challenged. 

Large language models hold immense promise and a glimpse into a future where understandable and supportive AI systems extend human capabilities. They will continue to enhance our decision-making and help us create content more quickly, automating mundane tasks so we can focus on more rewarding work.

However, as we harness LLMs for enhanced productivity, we must remain vigilant, understanding that, like any transformative technology, LLMs carry limitations and require responsible use. By leveraging the abilities of LLMs – while respecting and addressing their limitations with proactive strategies and innovative technologies – we can collectively steer a course toward a brighter, more equitable future.