AEGIS
Agent security evaluation harness
github.com/richter-max/aegis ↗A deterministic harness for measuring what a defense stack stops when a tool-using agent is attacked — and what it breaks in the process. 20 attacks across four families at three evasion tiers, plus 10 benign controls, run against every policy and guard configuration.
| GUARDS | ATTACK SUCCESS | FALSE POSITIVES |
|---|---|---|
| none | 100% | 0% |
| keyword | 70% | 20% |
| semantic | 70% | 0% |
| layered | 70% | 20% |
The harness runs a deterministic mock agent, not a live LLM. These figures characterize the guard stack — they are not an attack success rate for GPT‑4, Claude, or any real model.
Content inspection only catches attacks that name themselves.
Every guard stops 100% of the attacks that carry literal tokens like exfil, and 0% of the 14 that avoid the vocabulary. Read the 70% as one evasion tier fully stopped and two fully missed. There is no gradient in the data.
Layering made it strictly worse.
The layered stack detects exactly what the semantic guard detects on its own, and inherits the keyword guard's false positives on top. Two extra layers bought nothing. They cost 20% of benign traffic.
