As agents gain autonomy and tool access, safety becomes paramount. Guardrailing implements safety checks, input and output filters, and constraints to prevent agents from taking harmful or unintended actions. Stop conditions provide predefined criteria — task completed, max iterations reached, error threshold exceeded — that tell the agent when to stop iterating. The human-in-the-loop pattern requires human approval for certain decisions, balancing autonomy with safety, while the ask-before-acting pattern has the agent explain planned actions and wait for user confirmation. Progressive autonomy gradually increases an agent's freedom as trust is established, and confidence thresholds restrict autonomous action to cases where the agent's certainty exceeds a defined level.
The principle of least privilege dictates that agents should receive only the minimum permissions and tool access needed for their specific task, reducing risk of misuse. Sandboxing runs agents in isolated environments to limit damage from unexpected behavior, while capability control limits abilities like internet access or file deletion. The reversibility principle favors actions that can be undone when mistakes occur, and a kill switch provides immediate halt capability. Defense in depth layers multiple security measures — input validation, output filtering, sandboxing, monitoring — so that no single failure compromises safety. Structured agent output validation checks outputs against expected JSON schemas before processing.
Several well-known threats target agents. Prompt injection tricks the agent into ignoring its instructions, while indirect prompt injection embeds malicious instructions in external data sources the agent reads. The confused deputy problem arises when an agent with elevated permissions is manipulated into performing actions that benefit an attacker. Tool poisoning injects malicious instructions into tool descriptions or MCP server responses. Data exfiltration risk concerns agents being tricked into sending sensitive data to external endpoints. The OWASP Top 10 for LLM Applications catalogs these vulnerabilities, with excessive agency — granting agents too many capabilities, too much autonomy, or too many permissions — being a central concern. Defenses include input sanitization, instruction hierarchy enforcement, output filtering, separated data from instructions, and red-teaming to systematically discover vulnerabilities. Alignment research addresses deeper concerns: ensuring agents faithfully serve human interests (the principal-agent problem), remaining corrigible so they can be safely shut down, and avoiding specification gaming where they satisfy literal requirements while violating the intended spirit.