Testing APIs Against The OWASP LLM Top 10

Uploaded on 2024-11-08 in TECHNOLOGY-Key Areas-Artificial Intelligence, FREE TO VIEW, BUSINESS-Services-IT & Telecoms

GenerativeAI and Large Language Models (LLMs) are now being widely used in a business context which is significantly expanding the potential attack surface.

According to a Lightspeed survey, over 60% of those large businesses questioned said their business was using GenAI in some capacity in three or more use cases. Keen to harness the technology and gain competitive advantage, these businesses are rapidly integrating it into their operations and client facing offerings. But they’re doing so at such a rate that security is struggling to keep pace.

By 2025 its estimated there will be 750 million applications using LLMs but without sufficient testing and protection, these applications could be exposed to attack.

The focus in cybersecurity circles has been to use AI to protect against attacks but what if those attacks specifically target AI applications? It’s an issue that the OWASP industry group has grappled with, culminating in the release of the OWASP Top 10 for LLM Applications in August 2023. Aimed at developers, data scientists, and security experts tasked with designing and building applications and plug-ins leveraging LLM technologies, the Top 10 provides a concise but thorough overview of the main security issues affecting LLMs today. It covers attack tactics, techniques and procedures, with the top three attack types identified as prompt injection (both direct and indirect), insecure output handling and training data poisoning.

How Testing APIs Can Secure AI

Key to addressing the issue of securing these applications is testing the Application Programming Interfaces (APIs) they use to access data and to communicate with each other and to relay their findings. By using synthetic traffic to proactively testing LLM applications, it then becomes possible to identify vulnerabilities such as those highlighted in the OWASP Top 10. For instance, recent testing of several popular GenAI applications revealed indirect prompt injection vulnerabilities. When standard prompts were used, these did not yield a response, but malicious prompts were able to extract additional information from the AI systems.

In another example, it was revealed that Gemini for Workspace was susceptible to just such an attack back in September. By inserting malicious instructions into the data source the AI used, researchers were able to show they could manipulate the output of the Gemini LLM which the application integrates with.

Four tests were carried out.

The first saw an injected email sent to Gmail containing hidden instructions together with control tokens to trick the LLM into incorrectly summarising its content.

The second, similar in nature to a phishing attack, saw a warning to reset a password sent with a modified URL.

The third saw an attack carried out against Google Slides whereby speaker notes were used to prevent a correct summary being created.

While the fourth showed how sharing a file from Google Drive could be used to trick the LLM into sourcing and following instructions in a second document in the same shared folder.

Each of these attacks shows that even a heavyweight such as Google Gemini can succumb to prompt injection.

But the effects of insecure output handling, the second biggest threat identified in the OWASP list, can be just as damaging. It can lead to cross site forgery requests (CSRF) or remote code execution (RCE), as in the case of the Ollama vulnerability reported in June. That CVE saw Ollama provide insufficient validation, enabling an attacker to send a specially crafted HTTP request to its API. This could then be used to pull a model from Ollama and then supply a malicious file that contained a malicious payload, corrupting files held on the system and allowing the attacker to hijack it.

Preventing Poisoning

Third on the OWASP list is the poisoning of training data, which is typically seen as an open source issue, as illustrated by the one hundred malicious models that were found to have been uploaded to the Hugging Face AI repository by researchers in February. On another occasion, researchers found they could access Meta and Llama LLM repositories using unsecured API access tokens they discovered on GitHub and Hugging Face which would potentially have allowed them to poison the LLMs as well as steal models and data sets. However, this type of attack does not necessarily require a malicious actor, as models can theoretically succumb to degenerative model collapse and implode, thereby poisoning themselves. This highlights the need for both testing and detection to benchmark outputs and detect degradation in the quality of those.

What each of these attack types illustrates is the need for testing for such issues prior to deployment. By using an API-native solution, it’s possible to test AI applications against the OWASP Top 10 to determine if these can be subverted. Organisations can then provide recommendations to their developers who can take the necessary corrective action.

So far, the biggest threats posed against LLMs are the subversion of AI by malicious actors or through ungoverned or shadow AI. These issues all have one thing in common: too much trust being placed in the technology and not enough checks and balances being in put in place.

By using API testing on AI applications, it becomes possible to mitigate these risks and to prevent AI from going rogue.

Andy Mills is VP of EMEA for Cequence Security

Image: Allison Saeng

You Might Also Read:

The Unique TTPs Attackers Use To Target APIs:

If you like this website and use the comprehensive 6,500-plus service supplier Directory, you can get unrestricted access, including the exclusive in-depth Directors Report series, by signing up for a Premium Subscription.