Moviwa

Security of AI systems and autonomous agents

Securing artificial intelligence systems bears little resemblance to classical information security, and that difference is what catches organisations off guard. A language model does not distinguish between the instructions its developer gave it and the text that reaches it inside a document, an email or a web page. That single characteristic opens an entire family of attacks with no equivalent in traditional software.

Prompt injection is the best known case: place instructions inside content the system is going to read and it follows them as though they came from whoever operates it. When the system is an assistant that only drafts text, the damage is limited. When it is an agent with permissions — able to query databases, send email, execute actions or call other tools — the damage stops being hypothetical, because the attacker inherits the agent's permissions.

On top of that sits an architectural problem: agents are deployed with more autonomy and less human oversight all the time, precisely because their value lies in not having to approve every step. Each layer of autonomy added reduces the number of points where a person could stop something that has gone wrong, and the incidents documented across the industry show that this is the pattern that costs the most.

The controls that work are the familiar ones, applied to a new object: least privilege for the agent, separation between the content it reads and the instructions it executes, validation of outputs before they have effects, logging of all activity, and a human stopping point on irreversible actions. And, on the everyday-use side, a control over what information leaves for systems the organisation does not control.

This category holds the technical analyses, the demonstrations and the real incidents from the industry. If what you need is how information is protected before it leaves, the reference is the security page.