defending-against-prompt-injection

Defending against prompt injection

Source: Mitigate jailbreaks and prompt injection and Building trustworthy agents. The sandboxing skill caps what an attack can reach; this skill keeps the model from obeying the attack in the first place. The two stack: one deterministic boundary around the agent, one discipline for how third-party text enters its context.

An agent that reads outside text inherits a new attacker: whoever wrote the page, the email, or the API response it fetches. That text can carry instructions aimed at the model, and a helpful model follows them unless you arrange the context so it can tell your instructions from the data. The steps below lower the odds it confuses the two.

Keep untrusted content in tool_result blocks

Put every piece of third-party text in a tool_result block. Instruction-tuned models weight instructions inside a tool result below those in the system prompt, so a sentence that would hijack the agent from the system prompt carries far less weight there. The converse holds too: do not put your own instructions in a tool result, because the model discounts those as well. Send your instructions in a user turn after the result lands.

Wrap the content as data

Encode untrusted strings as JSON with explicit fields, such as {"source": "inbound_email", "from": "...", "body": "..."}. The escaping gives a hard boundary the attacker cannot step across: a line that tries to open a new instruction stays a string value inside body. Raw concatenated text has no such edge, so a crafted line reads as a fresh command.

Label where each block came from

Tell the model what a block is and who produced it. "The following is the body of an email from an unverified sender" sets how much trust to extend. An unlabeled block reads as authoritative by default.

Screen tool output before the agent acts