Agent hijacking: The true impact of prompt injection attacks

Agents in Large Language Models (LLMs) are computer programs that can make autonomous decisions or perform actions on behalf of a user or another system. They offer a flexible and convenient way to connect multiple application components, such as data stores, functions, and external APIs, to an underlying LLM. However, agents also pose additional risks, including the vulnerability to prompt injection attacks, which can be exploited by attackers using old and new techniques. Prompt injection is a variant of injection attack where user-provided input is reflected directly into a format that the processing system cannot distinguish between what was provided by the developer and the user. Successful prompt injections in agent-based systems can have an array of potential impacts, but this is typically constrained to the LLM itself. Classic vulnerabilities in AI agents are also present, including software vulnerabilities that have been around for decades and will continue to be around in the future. To address these risks, prompt defenses are required to identify and prevent prompt injection attacks and other AI-specific vulnerabilities in any LLM input or output.