llm-app-security

LLM Application Security

Rules (for AI agents)

ALWAYS

• Start from the premise that prompt injection is not solved at the prompt layer. A model has no privileged channel: system text, user text, a retrieved document and a tool result all arrive as one context, and nothing in that context can be marked as instructions the model must obey over the rest. Message roles are a formatting convention, delimiters raise the cost of an attack, and an instruction such as "treat the following as data" is itself just more text in the same window. Design as though injection will sometimes succeed, and put the real controls after the model rather than in front of it.
• Bound what the model's output is allowed to reach. Every downstream sink — a query builder, a shell, a file writer, an HTTP client, a renderer — must validate model output exactly as it would validate a request body from the internet, because that is what it is once injection is assumed. secure-code-review, database-security and frontend-security own the individual sinks; what belongs here is that model output never arrives pre-trusted.
• Constrain the output's shape with structured generation — a JSON Schema, function-call mode, constrained decoding — whenever the next step consumes it programmatically, and reject anything that fails validation rather than repairing it. A schema bounds the form, not the intent: a well-formed field can still carry a hostile value, so validate the value too.
• Authorize each tool call against the human user's permissions, checked at call time, not against what the model was told it may do. The tool allowlist is a capability list, not an authorization decision — and the agent's own privileges are the attack surface, because injected content drives the agent that holds them. Keep the agent's identity narrow for the same reason iam-best-practices keeps a pipeline role narrow.
• Put a human approval gate in front of consequential and irreversible actions: moving money, deleting or overwriting data, sending messages on a user's behalf, publishing, changing permissions. Injection that reaches a read-only tool is an information problem; injection that reaches a write tool is an incident. The gate is what keeps a single injected instruction from being final.
• Give retrieved content provenance and carry it through: which source, which tenant, which trust level. Retrieval is where the untrusted input usually enters, and the retriever is the last component that still knows where the text came from.
• Keep secrets, credentials, and internal hostnames out of the system prompt. Prompt-extraction is reliable enough to treat the system prompt as public, so it can hold instructions but not material whose value depends on staying hidden.
• Validate model parameters server-side — model name, system prompt, tool list, temperature. A client that can choose them can downgrade to a weaker model, replace your instructions, or widen the tool set.
• Bound the cost: per-tenant token budgets, request quotas, a cap on agent iterations and tool calls per task, and a wall-clock timeout. An agent loop with no ceiling is a billing incident that an attacker can trigger deliberately. api-security owns the endpoint's rate limiting.
• Record what was sent and what came back — prompt, model and version, retrieved context, tool calls — for audit, under logging-security's redaction policy. Without the retrieved context, an injection incident cannot be reconstructed.
• Consult ml-security before loading a model artifact — a checkpoint, an adapter, a fine-tune pulled from a hub. That skill owns the artifact: its format, its provenance, and the fact that loading a pickle-backed checkpoint executes code. This skill covers the application that calls a model, not the file the model arrives in.