Closing the Loop: A Self-Evolving Defense

Defender catches prompt injection in tool results. The previous two posts cover what it is (Training a 22MB Prompt Injection Classifier) and how we calibrated its false positive rate on real connector data (Detection Isn’t the Hard Part). This post is about the part neither covered: keeping it current.

A classifier ages. Prompt injection, indirect injection, and jailbreak techniques publish on Hugging Face, arXiv, and GitHub every week, and a static training corpus drifts a little further out of date with each one. Closing that gap manually meant a person reading a paper, hand-authoring attack samples, running an eval, and rebuilding the training set. The bottleneck wasn’t the model; it was attention.

This post is about Adaptive Defender, the system that runs that loop for us on a schedule, with optional human checkpoints along the way and a pull request at the end.

What it does

Once a week, Adaptive Defender does four things in order:

Discovers new attack techniques across Hugging Face, arXiv, and GitHub
Scores each one against both a panel of frontier models and the shipped Defender classifier
Proposes the corpus updates that close confirmed gaps
Retrains the classifier and files a pull request on the Defender repository

In auto mode the pipeline runs end-to-end from discovery through corpus integration with no operator input. The retrain itself is the one step that always sits with an operator: that’s where a human chooses whether to spend the GPU budget and ship a new model. Two checkpoints further upstream, approving the discovered sources and approving the proposed corpus delta, exist for operators who want them, but both are optional. Defender’s training corpus tracks new attack techniques on the same cadence the techniques are published.

What’s actually new

The pieces aren’t novel on their own. There’s an active ecosystem of red-team tooling. What’s different is the closure.

garak (NVIDIA) is a vulnerability scanner that runs 50+ probes for prompt injection, jailbreaks, and encoding attacks against an LLM. It runs known attacks; it doesn’t discover new ones or feed back into a classifier. PyRIT (Microsoft) is a red-team orchestration framework with multi-turn attack strategies and a web UI. Same shape: it runs attacks, but doesn’t close a defender loop. OpenAI Evals is a general benchmark harness, not red-team-specific. The closest research artifact is AutoRedTeamer (Zhou et al., 2025), which has a “strategy proposer” agent that mines recent literature for new attacks. But it’s an attacker targeting HarmBench success rates, not a system that closes into a shipping defender.

Every one of these stops at a stage. Adaptive Defender automates discover (new techniques from the wild) → dual-eval → corpus update → classifier retrain → PR on the shipping defender, end to end. That whole-loop closure is the contribution. Each piece on its own is familiar; wiring them into a single durable, scheduled pipeline that ships a PR isn’t.

The dual-eval quadrant

The core of the loop is how it decides which discovered samples matter. For every candidate attack we run two checks in parallel.

(A) Model panel. Does the attack actually exploit a frontier target model? The poisoned tool-result text gets fed to the model inside a realistic prompt, and a separate LLM judge labels exploited vs. resisted.

(B) Defender. Does the shipped classifier catch it? The real @stackone/defender package scores the same text. Each miss reflects production behavior, not a benchmark artifact.

The two signals answer different questions. The model panel is the severity filter: a sample that nothing real falls for isn’t an interesting attack, no matter how clever it looks on paper. Defender is the coverage signal: a sample we already catch is good news, not a training target. The loop cares about the intersection: a real exploit that Defender currently misses. Those become high-priority hard positives in the corpus.

The same eval pass also surfaces the inverse: benign payloads Defender flags as injection-shaped. Those become hard negatives, the same pattern from Detection Isn’t the Hard Part. One eval pass, two kinds of training signal.

Agentic discovery across the wild

Discovery is the only stage where the inputs are open-ended. Eval and retrain take a fixed list of samples; discovery has to find the list.

We do it with a stateful agent (a Cloudflare Agents SDK Durable Object) that reads the sources a security researcher would. Newly published attack and jailbreak datasets on Hugging Face. New technique papers on arXiv. Exploit-demo repositories on GitHub. The agent runs a ReAct loop with four tools: search, open, record, and done. It must actually open and read a candidate before recording a technique, and every recorded technique has to come with a concrete example payload, not abstract claims about what an attack “might” look like.

Because the agent is stateful, it remembers what it’s already seen. Schema mappings for HF datasets get cached per dataset; recorded techniques get embedded (Workers AI BGE) and deduplicated against the existing corpus. A second weekly run doesn’t re-propose what the first one already covered. The cache is what turns “an agent that explores the web” into “an agent that incrementally extends a corpus.”

The two source shapes complement each other. Newly published HF datasets give us breadth: real attack rows we can harvest in bulk. arXiv and GitHub give us depth: fewer samples, but novel techniques the model has to learn rather than recognize.

The loop

A discovery pass produces candidate techniques. From there it’s a single durable workflow:

Adaptive Defender pipeline: discover → curate (agent) → optional source-approval checkpoint → harvest + dual-eval per sample → integrate → optional corpus-approval checkpoint → retrain on Modal GPUs → PR on the Defender repo. Both checkpoints are configurable; when enabled they're durable wait-for-event steps with a 7-day budget that consume zero compute while paused.

The optional checkpoints are a Cloudflare Workflows primitive: each is a step.waitForEvent with a 7-day timeout. When enabled and paused, the workflow consumes no compute. It hibernates until the dashboard dispatches the approval event, then resumes mid-pipeline exactly where it left off. Run state, the per-sample eval results, the proposed corpus delta: all of it survives the pause because Workflows treats waitForEvent as a first-class durable step.

That single property is the difference between “this should be a workflow” and “this should be a workflow on Cloudflare.” A run-to-completion workflow couldn’t sit paused for two days waiting for someone to read the proposed source list; it would burn compute holding state or it would lose state on a deploy. Durable hibernation lets the checkpoints be slow without being expensive, which is what makes leaving them in as an option (rather than ripping them out for the autonomous path) actually viable.

The checkpoints exist so an operator who wants to review can review. In auto mode they’re skipped: discovery, dual-eval, and corpus integration all run end-to-end without operator input. The retrain at the end stays operator-controlled, because that’s where the GPU spend and the shipped-model change live. Adaptive Defender supports two promotion modes today: human_gated (both checkpoints enabled) and auto_corpus (gaps auto-added, retrain still operator-controlled). Most runs use the latter.

The PR

Once the corpus is finalized (either by an operator approving the delta, or by auto mode applying it), the workflow fires a Modal GPU job that retrains the classifier on the updated corpus, runs the full benchmark suite against the new weights, and, if the operator chooses to promote, opens a pull request on the Defender repository swapping in the new model. Promotion is gated on holding the calibration-set FPR from the prior post: a retrain that regresses on benign payloads doesn’t ship, regardless of what it gains on the new discovery samples. The PR shows up on GitHub, the same way a human-authored change would. It links back to the originating run; the eval results sit in the PR body; the new ONNX weights are in the diff.

That’s the loop closing on itself. A weekly run discovers attacks across HF/arXiv/GitHub, scores them through the quadrant, builds a corpus delta from confirmed gaps and FP risks, retrains the classifier, and ends as a PR against the package this post is about. It’s not a metaphor. The PR is the artifact.

Current state

The loop closes everywhere it’s cheap to be wrong, and stays gated at the one step where it isn’t. Discovery and dual-eval can be wrong cheaply: a bad sample wastes some Modal time, gets dropped at the quadrant stage, or shows up as a hard negative instead of a hard positive. Retrain is where the GPU spend and the shipped-model change live, so it stays operator-controlled. The upstream review checkpoints are optional and configurable per run.

The same shape handles the most obvious threat to a pipeline that ingests from public sources: someone publishing crafted datasets or repos to push hard positives that carve a blind spot, or benign-shaped payloads that inflate false positives. The dual-eval quadrant is the first filter: a planted sample that no frontier model falls for gets dropped as low-priority, and a benign-flagged payload becomes a hard negative, not a hard positive. The operator-reviewed retrain is the second filter, against the calibration set. Auto-mode handles the cheap-to-be-wrong stages; the expensive-to-be-wrong stage stays gated.

Numbers from individual runs are informative for tuning but aren’t a benchmark we publish; the post deliberately leans on architecture rather than headline metrics, because the loop’s value is what it does over time, not what any one run produces.

What’s next: more source coverage on the discovery side (better arXiv and GitHub recall), the deeper agentic exploit tier for severity, and a tighter token budget so weekly runs stay cheap. The hardest open problem is the one we flagged at the end of Detection Isn’t the Hard Part: payload shapes (and attack patterns) we’ll see for the first time in production. The loop helps because the moment a new shape does show up in the wild, it’s a candidate the discovery agent picks up, the quadrant scores, and the corpus learns to cover.

Each round had its hard part: detection in the first post, calibration in the second. This one’s about the part you can’t solve in a single pass: staying current.

Defender is open source: github.com/StackOneHQ/defender. Adaptive Defender is the system that keeps it current.