Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.
recordings/C2.16.mp4A patch that silences the scanner is indistinguishable from a patch that fixes the bug.
Stage 14: generate the fix, re-run the exploit against the patched build, and require a regression test.
Validate four candidate patches on three axes and show which of them only made the scanner green.
A patch that passes the tests and changes the behaviour is not a fix, it is a second incident with a pull request attached. Remediation is the stage where the pipeline stops finding things and starts touching them.
appsec/remediate.py, and the fix for row 1 of LABELS.md — which does not sanitise the id, because the id was never the problem. accept() requires three things and the scanner is not one of them; the third is a regression test that fails against the unpatched code. Hand it a test that passes both ways and it is refused, which is the most common thing a generated fix ships with.The stage whose wrong outcome looks exactly like the right one. Three pieces of evidence for a patch, none of them the scanner going quiet, and the one that is usually skipped: a regression test that fails against the unpatched code.
A patch that silences the scanner is indistinguishable from a patch that fixes the bug.
Generate the fix, re-run the exploit against the patched build, and require a regression test that fails on the old one.
Patches where the exploit no longer succeeds and the regression test fails pre-fix. Anything else is a silenced scanner.
Measure this
| what to measure | denominator or target |
|---|---|
| patches where the exploit no longer succeeds | share of patches |
| regression tests that fail pre-fix | share; anything else is a silenced scanner |
patch what has to be true +----------------+ +-----------------------------+ | fixes the bug | and | behaviour unchanged | | | | tests still pass | | | | reviewer can follow the why | +----------------+ +-----------------------------+ a patch that passes the tests and changes the behaviour is a second incident with a pull request attached
Stage 14 — Remediation engineering. Generate the fix, prove it, and only then ask anybody to look at it.
This stage comes after triage on purpose. Remediation is the first stage that touches the codebase, and touching it in the wrong order is expensive: fixing by rule severity means fixing the finding at the top of a queue that C2.15 has just told you is sorted wrongly. Calibrate first, then remediate, and the fix you write first is the one that matters most.
A model that finds bugs is useful. A model that fixes them is only useful if you can tell a real fix from a plausible one, and plausible is exactly what language models are optimised to produce.
There are three ways to make a finding stop firing:
All three make the scanner green, and an autonomous loop optimising for a green scan will find options 2 and 3 on its own because they are cheaper.
The pipeline has two things a static workflow does not, and both were built earlier in this chapter: Phase 4 produced a working exploit, and C2.6 built a replica to run it against. So a patch has somewhere to be proven before a reviewer's time is spent on it, and the promotion path is three environments with a different question at each:
| where | the question | what it catches |
|---|---|---|
| sandbox replica | does the exploit still work? | the patch that only moved the pattern |
| QA | does the application still do what it did? | the patch that closed the bug by breaking the feature |
merge request to master | is this the mechanism we want? | the design question no gate can answer |
The order is not arbitrary. The exploit re-run is first because it is the only acceptance test that cannot be satisfied by editing the code around the detector, and it is the one that gets skipped, always with the same justification: the fix is obvious.
One run that sounds redundant and is not: the exploit must also be re-run against the unpatched build in the same harness. Without it you cannot tell "the fix worked" from "the probe broke", and a probe that has quietly stopped asserting reports every candidate as a success.
Where you are in the pipeline. `` [Ingestion & Mapping] ──> [Threat Modelling] ──> [Discovery] └─ stages 1-4 └─ stages 5-6 └─ stages 7-10 ──> [Dynamic Validation] ──> [Reporting] └─ stages 11-14 └─ stage 15 ``Several candidate patches make the scanner green; one of them is a fix. The skill runs all three gates — behaviour unchanged, exploit blocked, and proof of fix against the old build — and reports which gate each rejected candidate died at.
Validation says a patch is good. It does not say the patch may be merged, and those are different decisions taken in different places. This skill runs the promotion path: the exploit re-run in the replica, the regression suite in QA, the finding diff in both directions, and only then a merge request against master with the evidence attached to it.
The same four candidates, walked through three environments. Watch what CI alone would have merged against what survives the gates — and then watch candidate D pass every one of them.
Candidate C is the argument for the replica. It is green, the rule no longer fires, and the exploit works unchanged — escape() moved the pattern past the detector without closing the injection. Nothing but re-running the exploit finds that, and nothing but an environment built to be attacked lets you.
Candidate D is the argument for the reviewer. It passes every gate: the exploit is blocked, the regression suite is green, no new findings. It closes the bug by returning an empty list, deleting partial-reference search — a feature travellers use. No automated gate rejects it, and none should, because is this the mechanism we want is not a property of a test run.
That is the division of labour the promotion path buys. The gates answer whether the bug is closed, on the page, before the request is opened. The reviewer's remaining job is D's question, which is the only one they were ever needed for.
The skill — skills/appsec/patch-validation-harness/SKILL.md
name: patch-validation-harness
description: >-
Accept a proposed fix only when three things hold — behaviour unchanged, the
exploit no longer works, and the fix proved against the build that was
vulnerable. Use when an agent proposes a patch, or when a scanner going green
is being read as a fix.
allowed-tools: Read, Grep, Glob, BashSeveral candidate patches will make the scanner green. Some of them change behaviour, some of them leave the bug exploitable, and one of them does neither. Telling them apart needs three gates, and the third is the one that is usually missing: proof against the old build, so "the exploit stops working" is a statement about the patch rather than about the environment.
Every proposed remediation, whether authored by a person or an agent, and especially when the evidence offered is that the scanner no longer fires.
1 — Establish the baseline on the vulnerable build. The behaviour cases must pass and the exploit must work. If the exploit does not work here, you are about to validate a patch against a bug you have not reproduced.
2 — Gate one: behaviour unchanged. Run every behaviour case against the patched build. A patch that fixes the defect and changes an answer is a regression with a security justification.
3 — Gate two: the exploit no longer works. Against the patched build, directly. Not "the scanner is quiet" — the scanner was one of the tools that missed the defect class in the first place.
4 — Gate three: proof of fix. Run the exploit against the old build again, after the patch is written, in the same harness. It must still work. This is what excludes the environment having changed underneath the test.
5 — Report per candidate and per gate. A candidate rejected at gate one and one rejected at gate two need different conversations with whoever wrote them.
Input — the fixture committed at the top of scripts/patch_validation_harness.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.
Output — the opening lines of a real run:
behaviour of the vulnerable build:
get_user('dana' ) → [(1, 'dana')] ok
get_user('sam' ) → [(2, 'sam')] ok
get_user('nobody' ) → [] ok
get_user("o'brien" ) → raised OperationalError FAILS
exploit works: True — probe returned 3 rows
scanner fires: FalseThe run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"baseline": {"behaviour_pass": true, "exploit_works": true},
"candidates": [{"id": "str", "scanner_green": true,
"behaviour_unchanged": true, "exploit_blocked": true,
"proof_of_fix": true, "verdict": "accepted|rejected", "rejected_at": "str|null"}],
"accepted": ["str"]
}The skill — skills/appsec/fix-promotion-gate/SKILL.md
name: fix-promotion-gate
description: >-
Decide whether a candidate security patch may leave the sandbox, pass QA and
become a merge request against the mainline. Use when an agent or a developer
proposes a fix for a confirmed finding, when a green CI run is being treated
as proof a vulnerability is closed, or when deciding what a security fix must
prove before review.
allowed-tools: Read, Grep, Glob, BashA green pipeline says the tests that exist still pass. For a security patch that is the wrong question, because the tests that exist were written by people who did not know about this bug — if they had, it would not be here.
There are three ways to make a finding stop firing and only one of them is a fix: fix the vulnerability, delete the feature, or evade the detector. All three turn CI green, and an autonomous loop optimising for green finds the second and third on its own, because they are cheaper.
The pipeline has something a static workflow does not: Phase 4 already built a working exploit, and C2.6 already built a replica to run it against. So the fix has somewhere to be proven before anybody is asked to look at it, and the promotion path is three environments with a different question at each:
| where | the question | what it catches |
|---|---|---|
| sandbox replica | does the exploit still work? | the patch that only moved the pattern |
| QA | does the application still do what it did? | the patch that closed the bug by breaking the feature |
| merge request | is this the mechanism we want? | the design question no gate can answer |
Skipping the first is the common failure, and it is always justified the same way: the fix is obvious.
On every patch for a confirmed finding, whether an agent or a person wrote it, before the merge request exists. Also when reviewing a remediation workflow that opens merge requests directly from a scanner going quiet.
1 — Re-run the exploit against the patched build, in the replica. Not the scanner. The exploit is the only acceptance test that cannot be satisfied by editing the code around a detector. A patch that leaves the exploit working has failed regardless of what the scan says.
2 — Prove the exploit still works against the unpatched build. Same run, same harness. Without this you cannot distinguish "the fix worked" from "the probe broke", and a probe that silently stopped asserting reports every patch as a success.
3 — Run the regression suite in QA, against real-shaped data. Behaviour preserved is a separate claim from vulnerability closed, and the patch that closes a SQL injection by returning an empty list satisfies the first gate perfectly.
4 — Diff the findings, both directions. The target finding must be gone and nothing new may appear. A patch that closes one injection and opens a path traversal has a net score of zero and a merge request that says "fixes CVE".
5 — Only now open the merge request, and put the evidence in it. The exploit run before and after, the regression result, the finding diff. A reviewer's remaining job is the question no gate can answer — whether this is the mechanism the codebase should use — and they can only get to it if the first three are already answered on the page.
Input — the fixture committed at the top of scripts/fix_promotion_gate.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.
Output — the opening lines of a real run:
the exploit, against the unpatched build in the replica
reproduces: True <- if this were False the probe is
broken and every result below would be meaningless
CI exploit behaviour new promoted to
A ok blocked same 0 merge-request
B ok blocked CHANGED 0 rejected <- qa: behaviour changed
C ok WORKS same 0 rejected <- sandbox: exploit still worksThe run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"candidates": [{"id": "str", "ci_green": true, "exploit_blocked": true,
"behaviour_preserved": true, "new_findings": 0,
"promoted_to": "rejected|qa|merge-request", "died_at": "str"}],
"would_merge_on_ci_alone": 0,
"merge_requests_opened": 0,
"evidence_attached": ["str"]
}would_merge_on_ci_alone against merge_requests_opened is the number this procedure exists to produce. If they are equal, the gates changed nothing and one of them is not actually running.
Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.
The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.
Install it, once. From the folder you cloned. --all does every agent it finds on your machine:
python3 scripts/install_skills.py --all --lessons C2.16Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.
Then pick c2-16-remediation-engineering:
| agent | install just for this agent | start it |
|---|---|---|
| Claude Code | python3 scripts/install_skills.py --tool claude --lessons C2.16 | type /c2-16-remediation-engineering |
| GitHub Copilot (VS Code) | python3 scripts/install_skills.py --tool copilot --lessons C2.16 | type /c2-16-remediation-engineering in Chat |
| Cursor | python3 scripts/install_skills.py --tool cursor --lessons C2.16 | type / in Agent chat and search for it |
| Codex | python3 scripts/install_skills.py --tool agents --lessons C2.16 | type $c2-16-remediation-engineering, or run /skills |
No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.
python3 scripts/lesson.py C2.16Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.
python3 scripts/checkpoint.py --at C2.16 --out work/cybertravels
python3 scripts/checkpoint.py --at C2.16 --diffOn Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.
Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.
Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.
If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.
The vulnerable build passes all four behaviour cases and the exploit returns 3 rows. Candidates A, B and D make the scanner green. Validation rejects B for changed behaviour and C for remaining exploitable, accepting A and D. Proof of fix holds for both accepted patches — the exploit works on the old build and fails on the new. The promotion path then walks the same four through the replica, QA and the finding diff: CI alone would have opened four merge requests and the gates open two, with C dying in the sandbox because the exploit still works and B dying in QA because the booking reference is silently truncated. D reaches review having passed every gate, which is the point — no automated check rejects a fix that works by deleting the feature.
Candidate D passes every automated gate and is still wrong. Write the rule that rejects it. You will find it has to be about which mechanism is acceptable, not about outcomes — and that rule belongs in your secure coding standard, not in the pipeline.
Tools used
Skill source: skills/appsec/patch-validation-harness