The increasing adoption of large language models (LLMs) like ChatGPT and Google Bard has been accompanied by rising cybersecurity threats, particularly prompt injection and data poisoning attacks. The U.K.’s National Cyber Security Centre (NCSC) recently released guidance on addressing these challenges.
Understanding Prompt Injection Attacks
Similar to SQL injection threats, prompt injection attacks manipulate AI outputs to produce unintended behaviors. These can range from displaying unethical content or malware to compromising the AI’s inner workings. For instance, a vulnerability in the LangChain library exploited by attackers was highlighted by NVIDIA’s Rich Harang. Another case saw MathGPT, which converts user inputs to Python code, being maliciously used.
This type of security concern isn’t just limited to backend systems. With chatbots becoming increasingly integrated into consumer-facing services, like online banking and shopping, there’s an inherent risk. Microsoft’s Bing Chat, for example, was shown to be vulnerable when a Stanford student, Kevin Liu, created an injection prompt revealing its internal script.
The Threat of Data Poisoning
Data poisoning represents another significant risk. It involves intentionally skewing the data sources feeding into machine learning models. With AI models scraping vast portions of the internet for training, they’re exposed to changing, inaccurate, or even malicious content. Research by teams from Google, NVIDIA, Robust Intelligence, and ETH Zurich unveiled two poisoning techniques: ‘split view data poisoning’, which takes advantage of ever-changing internet content, and ‘front-running attack’, exploiting temporary content edits on platforms like Wikipedia.
NCSC’s Recommendations
To combat these vulnerabilities, the NCSC emphasizes a comprehensive security approach:
1. Rigorous Input Validation: Consistent validation and sanitization to counter malicious inputs.
2. Safeguard Against Unsafe Libraries: Opting for safer serialization formats and avoiding inherently insecure ones like Python’s Pickle library.
3. Trust Verification: Ensuring data and software packages come exclusively from verified and trusted sources.
The central message from the NCSC is that as businesses and services become more reliant on AIs and chatbots, there is a paramount need to prioritize rigorous cybersecurity measures.
Keep an eye on our information security news updates as we continue to monitor “Cybersecurity Concerns In AI” and check the security experts respond to this news.
The latest NCSC guidance is rightfully suggesting the need to ‘exercise caution’ when building Large Language Models (LLM), with the explanation that our understanding of LLMs is still ‘in beta’ mode. As an industry, we are becoming more accomplished at using and making the most of the benefits of LLM, but there is more to learn about them, their full capabilities, and where their usage could leave individuals and indeed large organisations vulnerable to attack.
As organisations rush to embed AI into their applications, and startups begin to pop up with new and interesting ways to use this new form of AI; Language Models, such as OpenAI’s ChatGPT, it is important that developers understand how these models and their APIs work before building them.
Prompt Injection is currently the most common form of attack observed against LLMs, by focusing on defeating the protections they offer against sharing or creating information that could be damaging – for example, instructions on how to create malicious code. This is not the only danger, OpenAI has introduced “function calling”, a method for the AI to return data in a structured format that can be used by the application, making it easier for developers to expand on the AI’s capability or enrich its data with other sources.
The danger here is that those function signatures are sent to the AI in the same context, meaning that through prompt injection, attackers can learn the underlying mechanisms of your application and in some examples, attackers can manipulate the AI’s response to perform command injection or SQL injection attacks against the infrastructure.
To help raise awareness of this issue, Immersive Labs launched a “Beat the Bot” AI prompt injection challenge, available here ‘Immersive GPT’. In this challenge, users are tasked with building the right prompts to con the AI to give them the password. Of the 20,000 people that have attempted the challenge, around 3,000 made it through to level one, and only 527 made it to level 10, showing that there is still a lot for people to learn but even with varying levels of control it’s still easy to find a way to bypass a prompt.
By learning prompt injection, even your average person can trick and manipulate an AI chatbot. Real-time, gamified training becomes essential for not only attempting to keep up with the efforts of hackers, but also better understanding the ‘practice’ they are putting in themselves around AI prompt injection.