Safer Assistance

Agentic large language models (LLMs) offer a radically new approach to developing software by coordinating an entire ecosystem of agents in an imprecise conversation. This completely new way of working poses significant security risks, particularly due to manipulated prompts. A blog post [1] by renowned security specialist Bruce Schneier illustrates the problems: "We simply don't know how to defend against these attacks. We have zero agentic AI systems that are secure against these attacks. Any AI that is working in an adversarial environment – and by this I mean that it may encounter untrusted training data or input – is vulnerable to prompt injection. It's an existential problem that, near as I can tell, most people developing these technologies are just pretending isn't there."

To keep abreast of these risks, we comb through research articles written with a deep understanding of modern LLM-based tools and regularly summarize our findings in a blog [2], with the goal of providing an easy-to-understand practical overview of security issues and mitigations related to agent-based LLMs. Many risks are associated with agentic LLMs, and the technology is changing rapidly. It is therefore important to understand the risks and learn how to mitigate them whenever possible.

Agentic LLMs – a Definition of Terms

Artificial intelligence (AI) is changing rapidly, making its terms difficult to pin down. The term AI in particular is overused to cover everything from machine learning through LLMs to artificial general intelligence. The term agentic AI refers to LLM-based applications that can act autonomously. In order to act autonomously, AI agents extend the basic LLM model to include internal logic, loops, tool calls, background processes, and sub-agents. An agentic LLM taps a large number of data sources and can trigger activities with side effects (Figure 1).

[...]