Cyber Commons · Course guideA pipeline is a harness, and a harness needs a verifier
You already do this work: SAST, DAST, dependency and container scanning, and a penetration test before launch. Two things changed. Agentic code puts authority behind a model's output, so the classes that matter are ones deterministic rules cannot express — and the same coding agents now open pull requests faster than anyone can read them. The value of putting AI in the pipeline is not novelty; it is recall on the classes you were missing, at the rate you are now shipping.
Build the pipeline in order rather than buying it: threat model from what the estate already knows, deterministic SAST first and a model pass second, then deduplicate, filter for reachability, replicate in a sandbox, exploit dynamically, chain what composes, and only then calibrate severity and hand a fix to a human. Then the agentic penetration tests — white, black and grey box — and the controls an offensive agent has to run inside.
This function is measured on a before-and-after you can state out loud. Deterministic SAST on the sample in C2.3 has precision 1.00 and finds roughly one real defect in seven; the reasoning pass finds about six in ten — four times the recall, on the same code. That is the gain. The cost is false positives, which is what deduplication, reachability filtering and context engineering are for, and every one of those stages reports its own number rather than a claim.
Taught duration: about 5 hours (Function A's pace is 16 lessons in 4 hours)
Primary audience: Application security · AppSec and product security engineers · penetration testers
Prerequisites: Function A, which builds the system this function is about — at minimum A0.0 and A1.0
Reading: 14,022 words of lesson prose, at 180 words a minute.
One chapter and the whole function. Twenty lessons in pipeline order, from the threat model to the attestation, with the before-deploy and after-deploy split named in C2.0 and held for the rest.
Application security. AppSec engineers, product security engineers, secure code reviewers and penetration testers — the people who already run a security gate on every release, and now have to keep up with code an agent wrote.
Use this on your own, or to run Function C for other people. Everything here serves both; the shaded blocks marked For a room are the only parts a solo reader can skip.
Every per-lesson fact below is generated from the lesson sources and the skills — risk, control, the lab, Day 0/1/2, the measure, the command, and each skill's own committed example run. Unlike the Function A guide, it does not carry a hand-verified transcript of every run: sixteen of those were checked by running them, and writing 133 more without running them would be making them up. Trust the run over this page, and correct scripts/exercises/guides.py.
| On your own | Do it in pipeline order; the stages genuinely depend on each other and C2.4's deduplication is meaningless before C2.3 has produced duplicates. The two Bonus lessons (C2.14, C2.19) can be read at any point. |
|---|---|
| With a room | This is the function that works best with one shared screen and the pipeline drawn across a whiteboard, one stage added per lesson, because the value is the shape of the whole thing rather than any single stage. |
| Splitting it up | Anything over about four hours is normally split across sittings, and the chapter boundaries are the splits |
| The one sentence to hold | A pipeline is a harness, and a harness needs a verifier |
| Chapter | Lessons | What it does |
|---|---|---|
| C2 The AI SDLC — an Agentic AppSec Pipeline, Before and After Deploy | C2.0 · C2.1 · C2.2 · C2.3 · C2.4 · C2.5 · C2.6 · C2.7 · C2.8 · C2.9 · C2.10 · C2.11 · C2.12 · C2.13 · C2.14 · C2.15 · C2.16 · C2.17 · C2.18 · C2.19 | 20 lessons |
Half the time? C2.0, C2.1, C2.3, C2.4, C2.5, C2.15 and C2.16 — the harness argument, the two-pass scan, the two filters that make a queue workable, and the two that get a fix merged. That is the pipeline in miniature and it still has a verifier.
Run C2.10–C2.13 as one 20-minute discussion of what each colour of pentest can and cannot prove, rather than four labs.
The same three as every function. Way 1 is the default; the other two matter because a machine with no model configured can still do every lesson, and because "what actually changed?" is the question that comes up most.
| Way | What it does | Needs a model? | Use it for |
|---|---|---|---|
| 1 — in your AI agent | The agent reads the lesson, teaches it, runs the check | No — Claude is the agent | The default for every lab |
2 — scripts/lesson.py <id> | Runs the lesson, prints a five-heading readback | Yes, or it exits 2 saying "no model configured" | Worth doing once in A0.0, to meet that message on purpose |
3 — scripts/checkpoint.py --at <id> --diff | Shows the code this lesson added | No | Any "what actually changed?" question |
This function is about a system Function A builds. You do not have to have built it — a checkpoint writes it for you — but you do need it on disk.
python3 scripts/doctor.py, or python scripts\doctor.py on Windows.python3 scripts/checkpoint.py --at C2.0 --out work/c writes the tree with everything taught up to here and nothing taught after.python3 scripts/install_skills.py --all --lessons C.Read this before the first lesson. A taught session opens with it; alone, read it and move on.
One chapter, twenty lessons, one pipeline built end to end. C2.0 splits it the way the function is organised: what runs before a deploy, and what runs after. Everything else is one stage of that pipeline, in the order it runs.
The spine is: threat model from what the estate already knows → deterministic SAST, then a model pass → deduplicate and verify in context → filter to what is reachable → replicate in a sandbox → supply chain → dynamic exploitation → chaining → the agentic pentest in three colours (white, black, grey box) → severity and reporting → remediation proven before the merge request → context engineering to cut the false positives → attesting control intent.
The argument underneath all twenty is C2.1's: this is security engineering, so build a harness rather than a prompt. Every stage has a deterministic half and a model half, and the deterministic half is what makes the model half checkable. A pipeline whose only verifier is another model has no verifier.
Each lesson's risk, control, days, measure and skill — and, folded, what that skill prints.
One chapter and the whole function. Twenty lessons in pipeline order, from the threat model to the attestation, with the before-deploy and after-deploy split named in C2.0 and held for the rest.
Two of the twenty are marked Bonus — C2.14 on the controls an offensive agent runs inside, and C2.19 on the pipeline in production. Neither is optional reading if you intend to run an offensive agent against anything you do not own.
Twenty dependent stages is the longest single arc in the curriculum. Draw the pipeline once and add to it, rather than re-drawing per lesson.
A security pipeline built as if it were exempt from the risks it exists to find.
Build the pipeline and the harness as one system, and hold both to the same evidence standard.
| Day 1 — what you do |
|---|
| Learn the line first — what can run before a deploy and what only works after one — then build the pipeline and its harness as one system. |
Day 2 — measure. nothing is computed here (the output is the order the stages run in, which is what makes their numbers comparable)
The lab. Run a real LLM loop against a CyberTravels finding, then add the verifier and watch the same loop refuse what it just accepted.
sdlc-control-placementpython3 skills/appsec/sdlc-control-placement/scripts/sdlc_control_placement.pyIts output contract requires techniques, risks, counts, gate_capable. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
sdlc-control-placement prints — its own committed example runtechniques, placed
technique side gate blind to
SAST / code analysis pre yes whether the vulnerable path is reached with real traffic
SCA and SBOM scanning pre yes what actually loaded, versus what the manifest pinned
IaC and policy-as-code pre yes the identity and network the workload really got
threat modelling pre - entry points added by a config change, not a commit
tool-schema / MCP review both yes a description edited by the server after review
DAST / attack simulation post - the commit that introduced itA harness whose verifier is the model agreeing with itself does not fail loudly. It succeeds incorrectly, files a clean trace, and the bug is found by whoever merged the patch.
An independent verifier, and a budget that stops the loop when it cannot pass.
| Day 0 — why it matters |
|---|
| A harness whose verifier is the model agreeing with itself does not fail loudly. It succeeds quietly and wrongly. |
Day 2 — measure. agreement between the verifier and held-out ground truth (rate); loops stopped by budget rather than by success (count)
The lab. Run the same loop twice — once with no verifier, once with one.
agentic-harness-looppython3 skills/appsec/agentic-harness-loop/scripts/agentic_harness_loop.pyIts output contract requires loop, parse, verifier, runs. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
agentic-harness-loop prints — its own committed example runthe loop, wired: ['plan', 'act', 'verify', 'stop']
backend : replay
steps : 1
accepted : return DB.execute("SELECT * FROM bookings WHERE ref=?", (ref,))
verified : None <- nothing checked it
The loop stopped because the model produced something, which is not the
same as producing something correct. Whatever came back was accepted.Threat models are written once, by hand, against a system that has since changed.
Stage 5: derive assets, entry points and attack vectors mechanically from the synthesised map.
| Day 0 — why it matters | Day 1 — what you do |
|---|---|
| Threat models are written once, by hand, against a system that has changed since. | Derive assets, entry points and attack vectors mechanically from the map the estate already holds. |
Day 2 — measure. diff between two threat-model runs (count of changed rows; no diff when the estate changed means it is not being derived)
The lab. Turn an architecture map into a ranked threat model, then diff it after one entry point is added.
threat-model-stridepython3 skills/appsec/threat-model-stride/scripts/threat_model.pyIts output contract requires inputs_present, threats, boundaries, diagram, diff. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
threat-model-stride prints — its own committed example runid entry sink score why
--------------------------------------------------------------------------------------------
T-10 I upload_voucher store 11 a live CSPM finding on the resource this path reaches
T-01 S get_booking load_booking 11 the running role is assumable by *
T-05 D get_booking load_booking 10 internet-facing with no WAF in front of it
T-06 E get_booking load_booking 10 the identity holds write, not just read
T-03 R get_booking load_booking 10 no MFA on the assumable role, so the actor is not established
T-04 I get_booking load_booking 9 no open CSPM finding on the resourcegenerated-schema-auditpython3 skills/research/generated-schema-audit/scripts/generated_schema_audit.pyIts output contract requires statements, policies, catalogue, findings. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
generated-schema-audit prints — its own committed example runtables created : profiles, posts
findings : 2
table severity why
profiles critical RLS disabled - open via the public API
posts high policy matches every row
The application works. Every feature passes. `profiles` holds an api_keyPattern matching floods the queue; the false-positive rate is what actually changed.
Stage 7: deterministic rules for what rules do well, model reasoning for what rules cannot express.
| Day 0 — why it matters | Day 1 — what you do |
|---|---|
| Pattern matching floods the queue, and the classes that matter in agentic code cannot be written as patterns at all. | Deterministic rules for what rules do well, then a model pass for what they cannot express. |
Day 2 — measure. SAST precision and recall, deterministic pass (precision 1.00, recall about 1 in 7 on the sample); recall of the reasoning pass on the same code (about 6 in 10 — four times the recall, and the extra false positives are the next four stages' work)
The lab. Score grep, taint rules and model review against the same corpus, then combine them behind a confidence gate.
sast-semgrep-deterministicpython3 skills/appsec/sast-semgrep-deterministic/scripts/sast_semgrep_deterministic.pyIts output contract requires key, runs, missed_by_all, widest_recall. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
sast-semgrep-deterministic prints — its own committed example runsemgrep 1.176.0 · booking.py · 6 defects in the key, 5 of them expressible as a pattern
config found tp fp prec recall
p/python + p/secrets 1 1 0 1.00 0.17
seven registry packs 4 4 0 1.00 0.67
custom taint rule 2 2 0 1.00 0.33
Nothing about the file changed between those rows. Three of the fouridor-detection-recallpython3 skills/appsec/idor-detection-recall/scripts/idor_detection_recall.pyIts output contract requires units, detectors, findings, denominator. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
idor-detection-recall prints — its own committed example runcybertravels/ · 7 object-handling units, 4 of them authorised
detector found tp fp prec recall
pattern rule (execute + id) 0 0 0 0.00 0.00
ownership-comparison analysis 3 3 0 1.00 1.00sast-model-passpython3 skills/appsec/sast-model-pass/scripts/sast_model_pass.pyIts output contract requires backend, hypotheses, rejected, promoted. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
sast-model-pass prints — its own committed example runget_booking
model says : MISSING CWE-639 at confidence 0.82
quoting : def get_booking(session, booking_id):
-> HYPOTHESIS: quote verified, status hypothesis (not a finding)
get_my_booking
model says : PRESENT CWE-639 at confidence 0.77
quoting : require_owner(session, row["owner_id"] if row else None)
-> NO CLAIM : reviewed, nothing asserted, nothing to promoteParallel analysis tracks report the same bug three times, and some of those bugs do not exist.
Stages 8–9: consolidate overlapping findings, then cross-reference each one against syntax and imports to weed out hallucinations.
| Day 1 — what you do |
|---|
| Consolidate overlapping findings, then cross-reference each against syntax and call graph. |
Day 2 — measure. findings before and after consolidation (two counts, both reported); findings that survived verification (count)
The lab. Deduplicate findings across three analysis tracks, then verify each against the AST and drop the ones that reference code that is not there.
finding-dedup-and-verificationpython3 skills/appsec/finding-dedup-and-verification/scripts/finding_dedup_and_verification.pyIts output contract requires raw, defects, rejected, counts. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
finding-dedup-and-verification prints — its own committed example run8 raw findings from 3 tracks, against tools/bookings_api.py
grep CWE-89 tools/bookings_api.py:41 execute conf=0.50
taint CWE-89 tools/bookings_api.py:41 execute conf=1.00
model CWE-89 tools/bookings_api.py:42 execute conf=0.93
model CWE-943 tools/bookings_api.py:41 execute conf=0.71
model CWE-639 tools/bookings_api.py:20 get_booking conf=0.82
model CWE-89 tools/bookings_api.py:47 execute conf=0.44
model CWE-798 tools/bookings_api.py:8 DB_PASSWORD conf=0.88
model CWE-78 tools/bookings_api.py:34 os.system conf=0.67A finding in dead code costs the same to triage as one on the login path.
Stage 10: decide whether an external caller can actually reach the sink before anyone is paged.
| Day 0 — why it matters |
|---|
| A finding in dead code costs exactly as much to triage as one on the login path. |
Day 2 — measure. findings with a reachable path from a real entry point (share of findings); triage hours the unreachable findings would have cost (hours)
The lab. Build a call graph from entry points and partition findings into reachable, unreachable and unknown.
appsec-vuln-auditpython3 skills/appsec/appsec-vuln-audit/scripts/appsec_vuln_audit.pyIts output contract requires findings, dropped, counts. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
appsec-vuln-audit prints — its own committed example runcall graph:
http_get_report → ['load_report']
http_health → —
load_report → —
legacy_export → —
debug_dump → —
dispatch → —dead-code-ast-reachabilitypython3 skills/appsec/dead-code-ast-reachability/scripts/dead_code_ast_reachability.pyIts output contract requires graph, buckets, unresolved_calls, findings, queue. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
dead-code-ast-reachability prints — its own committed example runreachable list_reports, load reached from an entry point
unreachable legacy_export no caller, no dynamic call in api.py
unknown dispatch, nightly jobs.py contains getattr(...)()Dynamic testing is run against staging, so a destructive probe becomes an incident.
Stage 11: replicate the application in an isolated, disposable runtime with no path to production.
| Day 0 — why it matters |
|---|
| Dynamic testing run against staging turns a destructive probe into an incident. |
Day 2 — measure. probes executed in the sandbox (count); connections from the sandbox to anything production (count; must be zero)
The lab. Stand up an isolated replica, prove egress and credential isolation, and show what a destructive probe touches.
exploit-replica-isolation-checkpython3 skills/appsec/exploit-replica-isolation-check/scripts/exploit_replica_isolation_check.pyIts output contract requires environment, checks, verdict, safe_to_exploit. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
exploit-replica-isolation-check prints — its own committed example runALLOW http://replica.local/reports replica-internal
ALLOW http://db.replica.local:5432/ replica-internal
DENY https://api.github.com/ not on the replica allowlist
DENY http://169.254.169.254/latest/meta-data/ private address outside the replica — blocked
DENY http://10.0.3.14:9200/_search private address outside the replica — blocked
probe on staging on replica
--------------------------------------------------------
drop a table REACHES containedA clean dependency scan on an estate carrying an undeclared third-party binary reads as evidence of safety, and is evidence of nothing but the manifest's contents.
Reconcile the SBOM against what is on disk, then recover strings, imports and egress from the compiled artefact that no SBOM entry covers.
| Day 1 — what you do |
|---|
| Reconcile the SBOM against what is actually on disk, then recover strings, imports and egress from the binaries it missed. |
Day 2 — measure. artefacts on disk that appear in the SBOM (share; the gap is the finding and is usually not zero)
The lab. Scan an SBOM, then decompile the closed-source library it never mentions.
supply-chain-decompilepython3 skills/appsec/supply-chain-decompile/scripts/supply_chain_decompile.pyIts output contract requires sbom, advisories, reconciliation, decompiled, unassessable_by_sbom. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
supply-chain-decompile prints — its own committed example runSBOM: cybertravels-booking 4.7.1 · 5 declared components
CRITICAL org.apache.commons/commons-text 1.9 GHSA-599f-7c49-w659
Variable interpolation permits script execution (Text4Shell) -> fixed in 1.10.0
HIGH pyyaml 5.3.1 GHSA-8q59-q68h-6hv4
Arbitrary code execution via full_load / FullLoader -> fixed in 5.4
MEDIUM requests 2.31.0 GHSA-h5c8-rqwp-cp95
Proxy-Authorization header leaked on redirect -> fixed in 2.32.0A SAST finding is a hypothesis, and hypotheses get argued about instead of fixed.
Stage 12: generate and run an actual exploit against the sandbox, so the finding is confirmed or dropped.
| Day 0 — why it matters | Day 1 — what you do |
|---|---|
| A static finding is a hypothesis, and hypotheses get argued about in triage meetings instead of fixed. | Generate and run an actual exploit against the sandbox, so each finding is confirmed or dropped. |
Day 2 — measure. findings confirmed by execution, against those refuted (two counts; only one of them is a ticket)
The lab. Turn static findings into executable probes against the replica and separate confirmed from unconfirmed.
dynamic-exploitation-probepython3 skills/appsec/dynamic-exploitation-probe/scripts/dynamic_exploitation_probe.pyIts output contract requires hypotheses, probes, assertions, weak_assertion_run. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
dynamic-exploitation-probe prints — its own committed example runnormal use:
list_reports('test-user-a') → [(1, 'test-user-a', 100), (2, 'test-user-a', 250)]
fetch_doc('readme.txt') → public readme
probe cwe exploited observation
--------------------------------------------------------------------------
P-01 CWE-89 True 4 rows, owners=['test-user-a', 'test-user-b', 'test-user-c']
P-02 CWE-22 True returned 'synthetic-not-real-9999'
P-03 CWE-89 False 2 rows, owners=['test-user-a']Three medium findings are triaged as three mediums, and nobody notices they compose.
Stage 13: combine validated findings into multi-step sequences and score the chain, not the links.
| Day 1 — what you do |
|---|
| Combine validated findings into multi-step sequences where one's effect satisfies another's precondition. |
Day 2 — measure. chains discovered (count); chain severity against the highest severity of its links (delta; the difference is what individual triage missed)
The lab. Chain individually-medium findings into a critical path and show the severity the chain earns.
appsec-exploit-validatepython3 skills/appsec/appsec-exploit-validate/scripts/appsec_exploit_validate.pyIts output contract requires validations, refused. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
appsec-exploit-validate prints — its own committed example runid cwe sev requires grants
------------------------------------------------------------------------------------------------
F-01 CWE-200 low ['unauthenticated'] ['valid_report_id']
F-02 CWE-639 medium ['authenticated', 'valid_report_id'] ['other_users_report_data']
F-03 CWE-352 medium ['authenticated'] ['password_reset_for_victim']
F-04 CWE-89 high ['authenticated'] ['database_read']
F-05 CWE-306 medium ['internal_network'] ['admin_actions']
F-06 CWE-918 medium ['authenticated'] ['internal_network']Payload suggestions instead of attack chains — and an offensive loop with no hard scope enforcement, which is an incident with a project plan.
Full target context before it swings, and scope enforced at the network layer rather than by a politeness clause in the prompt.
| Day 0 — why it matters | Day 1 — what you do |
|---|---|
| An offensive loop with no hard scope boundary tests something you were not authorised to touch, at machine speed. | Give the loop full target context, and enforce scope at the network layer rather than by asking the model to respect it. |
Day 2 — measure. findings ranked against a severity-sorted baseline (rank correlation); out-of-scope requests the enforcement refused (count)
The lab. Drive a planner/executor pair against a local target and watch the scope guard refuse an out-of-scope host before the request leaves.
offensive-agent-containmentpython3 skills/redteam/offensive-agent-containment/scripts/offensive_agent_containment.pyIts output contract requires baseline, model_triage, adversarial, enforced, conclusion. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
offensive-agent-containment prints — its own committed example run=== generation 1: manual — a human reads all six and decides ===
6 findings, no ordering, ~20 min of reading
=== generation 2: scripted — sort by severity ===
F-02 high /v1/users returns data without auth
F-01 medium TLS 1.0 enabled
F-04 medium password auth permitted
→ severity is a label, not a prediction. F-04 and F-05 are bothFull source produces a finding list nobody can act on, because presence is reported where reachability was needed.
Every candidate carries the path that reaches it and the authorisation predicate on that path; unreachable sinks are reported as unreachable rather than dropped.
| Day 1 — what you do |
|---|
| Enumerate paths from real entry points to sinks, and classify the predicate on each hop as authentication or authorisation. |
Day 2 — measure. sinks reachable from an entry point (count — three of five on CyberTravels); reachable sinks carrying only an authentication check (count)
The lab. Enumerate paths from four entry points in the CyberTravels tree and separate reachable sinks from present ones.
whitebox-path-reachabilitypython3 skills/redteam/whitebox-path-reachability/scripts/whitebox_path_reachability.pyIts output contract requires data_sources, paths, counts, findings. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
whitebox-path-reachability prints — its own committed example run AUTHN ONLY POST /refunds handler -> svc.refund -> payments.refund
authn: session authz: NONE ON ANY HOP
sinks present in the tree 7
sinks reachable from an entry 5
of those, authenticated only 3An agent narrates a confident architecture from status codes, and the report is fiction that reads like findings.
Every claim is labelled observed or inferred, with the evidence that supports it, and inferences never carry a severity.
| Day 1 — what you do |
|---|
| Label every claim observed or inferred, keep the evidence beside it, and attach no severity to an inference. |
Day 2 — measure. findings against open questions (two counts — five and seven on the sample run); what each open question would need to resolve it (named, per question)
The lab. Score twelve claims from an external probe run and see which survive the observed/inferred split.
blackbox-claim-provenancepython3 skills/redteam/blackbox-claim-provenance/scripts/blackbox_claim_provenance.pyIts output contract requires data_sources, claims, findings, open_questions. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
blackbox-claim-provenance prints — its own committed example runreport: 5 findings, 7 open questions
no severity is attached to anything in the second list.Object-level authorisation is assumed correct because the endpoint list was covered, and BOLA lives in the cells nobody enumerated.
A roles-by-objects-by-verbs matrix with every cell marked tested, assumed or unreachable, and the assumed cells ranked by blast radius.
| Day 0 — why it matters | Day 1 — what you do |
|---|---|
| Object-level authorisation is assumed correct because the endpoint list was covered, and broken object access lives in the cells nobody enumerated. | Fill a roles-by-objects-by-verbs matrix from one credential per role, and mark every cell tested, mismatched or untested. |
Day 2 — measure. authorisation cell coverage, not endpoint coverage (12 of 30 cells on the sample); mismatches, and untested cells ranked by blast radius (count, and the two highest-cost cells)
The lab. Fill the matrix for CyberTravels with three roles and find the untested cells that matter.
greybox-authorization-matrixpython3 skills/redteam/greybox-authorization-matrix/scripts/greybox_authorization_matrix.pyIts output contract requires data_sources, cells, mismatches, untested, coverage. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
greybox-authorization-matrix prints — its own committed example runtested 12 of 30 cells (40%)
2 mismatch(es):
! traveller can read booking(other): designed deny, observed allowThe offensive agent is the least supervised and most capable thing in the estate, and its own traffic looks exactly like an attack.
A preflight that refuses to start the engagement until every control is present, and tells the SOC what to expect.
| Day 0 — why it matters | Day 1 — what you do |
|---|---|
| The offensive agent is the most capable and least supervised thing in the estate, and its traffic is indistinguishable from an attack by design. | Run a preflight that refuses to start until zero retention, sandboxing, egress control, secret management, human-in-the-loop and deterministic guardrails are all present. |
Day 2 — measure. blocking controls present (count out of six); whether the SOC was briefed (yes/no; the refusal is the deliverable when it is no)
The lab. Run the preflight against two engagement configurations and read why one of them cannot start.
offensive-agent-safety-preflightpython3 skills/redteam/offensive-agent-safety-preflight/scripts/offensive_agent_safety_preflight.pyIts output contract requires controls, engagements, soc_brief. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
offensive-agent-safety-preflight prints — its own committed example run ENGAGEMENT REFUSED — 4 blocking control(s) missing:
zero_data_retention, egress_allowlist, human_in_the_loop, deterministic_guardrailsSeverity is a label copied from the rule, so the queue is ordered by something that predicts nothing.
Stage 15: calibrate severity from sandbox evidence, then report per-stage economics rather than a finding count.
| Day 0 — why it matters | Day 1 — what you do |
|---|---|
| Severity copied from the rule orders the queue by something that predicts nothing. | Calibrate severity from sandbox evidence, and report per-stage economics rather than a single total. |
Day 2 — measure. cost per confirmed finding, per stage (currency per finding; it is what says which stage to invest in next)
The lab. Recalculate severity from confirmed exploitation and reachability, then produce the per-stage escape economics.
appsec-triage-reportpython3 skills/appsec/appsec-triage-report/scripts/appsec_triage_report.pyIts output contract requires report. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
appsec-triage-report prints — its own committed example runid rule sev calibrated score
----------------------------------------
F-01 high critical 7 ← moved
F-02 high low 0 ← moved
F-03 medium critical 6 ← moved
F-04 high medium 2 ← moved
F-05 medium critical 6 ← moved
F-06 high medium 2 ← movedA patch that silences the scanner is indistinguishable from a patch that fixes the bug.
Stage 14: generate the fix, re-run the exploit against the patched build, and require a regression test.
| Day 1 — what you do |
|---|
| Generate the fix, re-run the exploit against the patched build, and require a regression test that fails on the old one. |
Day 2 — measure. patches where the exploit no longer succeeds (share of patches); regression tests that fail pre-fix (share; anything else is a silenced scanner)
The lab. Validate four candidate patches on three axes and show which of them only made the scanner green.
patch-validation-harnesspython3 skills/appsec/patch-validation-harness/scripts/patch_validation_harness.pyIts output contract requires baseline, candidates, accepted. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
patch-validation-harness prints — its own committed example runbehaviour of the vulnerable build:
get_user('dana' ) → [(1, 'dana')] ok
get_user('sam' ) → [(2, 'sam')] ok
get_user('nobody' ) → [] ok
get_user("o'brien" ) → raised OperationalError FAILS
exploit works: True — probe returned 3 rows
scanner fires: Falsefix-promotion-gatepython3 skills/appsec/fix-promotion-gate/scripts/fix_promotion_gate.pyIts output contract requires candidates, would_merge_on_ci_alone, merge_requests_opened, evidence_attached. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
fix-promotion-gate prints — its own committed example runthe exploit, against the unpatched build in the replica
reproduces: True <- if this were False the probe is
broken and every result below would be meaningless
CI exploit behaviour new promoted to
A ok blocked same 0 merge-request
B ok blocked CHANGED 0 rejected <- qa: behaviour changed
C ok WORKS same 0 rejected <- sandbox: exploit still worksThe model is given the repository and asked to be thorough, so the relevant line falls out of the window.
Slice on the source-sink path, not on distance: the smallest context that still supports a severity decision.
| Day 0 — why it matters | Day 1 — what you do |
|---|---|
| The model is handed the repository and asked to be thorough, so the relevant line falls out of the context window. | Slice on the source-to-sink path rather than on distance: the smallest context that still supports a severity call. |
Day 2 — measure. false positives before and after slicing (two counts); tokens spent (count; both should improve together)
The lab. Compare four context strategies against one bug and measure which are decidable and at what size.
context-window-sizingpython3 skills/appsec/context-window-sizing/scripts/context_window_sizing.pyIts output contract requires decidability, slices, chosen, excluded. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
context-window-sizing prints — its own committed example runthe bug is on line 22
whole file 839 chars 29 lines
±2 line window 207 chars 5 lines
±6 line window 432 chars 13 lines
strategy sink concat source intent decidable
----------------------------------------------------------------
whole file True True True True True
±2 line window True True False False FalseControl claims are asserted in a spreadsheet and never bound to a deployment. Nobody can say which repo, image, role, identity, gateway and guardrail the claim was about, so it cannot be re-checked when any of them change.
Eleven skills scoped to one deployment_id, emitting an in-toto/DSSE attestation whose predicate carries per-control verdicts, evidence URIs, framework mappings and drift — with sandbox-egress and injection-screening capped at PARTIAL because their claims are not provable.
| Day 0 — why it matters | Day 1 — what you do |
|---|---|
| Control claims live in a spreadsheet and are never bound to a deployment, so nobody can say which build they describe. | Scope every skill to one deployment identifier and emit a signed attestation whose predicate names what was checked. |
Day 2 — measure. deployments with a verifiable attestation (share of deployments); claims with no attestation behind them (count)
The lab. Run the control-intent analyser over ten real agent and MCP repositories and read the attestation it produces for each.
agent-code-surface-analyzerpython3 skills/attestation/agent-code-surface-analyzer/scripts/agent_code_surface_analyzer.pyIts output contract requires deployment_id, tools, dangerous_actions, unannotated_count, verdict. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
agent-code-surface-analyzer prints — its own committed example runC1_default_deny_least_privilege INTENT_EVIDENCED runtime verdict needs observed usage and the
C2_sandbox_no_egress PARTIAL absence of a covert channel is not provable
C3_identity_chain_obo INTENT_EVIDENCED 4 signals; enforcement not shown
C4_gateway_guardrails INTENT_EVIDENCED runtime verdict needs reachability testing f
C5_injection_screening PARTIAL detector presence is verifiable; robustness
PASS is not in the vocabulary. The strongest static verdict is
INTENT_EVIDENCED, and two controls cannot exceed PARTIAL at all.A reference implementation is adopted as a product, and its outputs are trusted without an eval.
Map Mantis's stages onto the pipeline you built, then score it with your own held-out key before trusting it.
| Day 0 — why it matters | Day 1 — what you do |
|---|---|
| A reference implementation gets adopted as a product and its outputs trusted without an evaluation. | Map its stages onto the pipeline you built, then score it against your own held-out key. |
Day 2 — measure. a harness's accuracy on your key (rate — not its accuracy on its own key, which does not transfer)
The lab. Map Mantis onto the 15 stages, parse its two output shapes, and score a sample against a held-out key.
reference-pipeline-scoringpython3 skills/appsec/reference-pipeline-scoring/scripts/reference_pipeline_scoring.pyIts output contract requires stages, uncovered_phases, conformance, score, errors. The script validates the reply against these, so a run that answers in the wrong shape fails rather than passing quietly.
reference-pipeline-scoring prints — its own committed example runstage name mantis note
------------------------------------------------------------------------------------------------
1 historical parsing yes historical_learnings.jsonl is read on subsequent runs
2 structural indexing partial operates over the agent's code-reading tools
3 component summarisation partial context assembled per review target
4 architecture synthesis no assumes you supply the architecture context
5 threat modelling partial review skills encode threat patterns rather than deriving them
6 strategic planning no you decide what to point it atcybertravels/ and compare your findings with cybertravels/LABELS.md. That file is the hand-written key four of these skills score recall and precision against. Report both, and report what you missed.Have each pair report recall against LABELS.md without saying which stage they trusted — then reveal both. The correlation is the lesson.
Cover the answer column, or use any three with a room.
| Question | Answer |
|---|---|
| 1. What makes a security pipeline a harness rather than a prompt? | A deterministic half that the model half is scored against |
| 2. Why does the deterministic scan run before the model pass? | So the model pass has something to be measured against |
| 3. C2.4 and C2.5 both shrink the queue. What is the difference? | One removes duplicate reports of one defect; the other removes defects in code nothing calls — only the second is a statement about risk |
| 4. What must a remediation test do before a fix can be merged? | Fail against the unpatched code |
| 5. A black-box probe reports a finding confidently. What do you ask? | Which part the evidence entails and which part it merely suggests |
| 6. Conformance is near 100%. What does that tell you about quality? | Nothing. Conformance is schema validity; accuracy is correctness |
| What you built | Goes on to | Where it is taken apart |
|---|---|---|
| The pipeline you built | D1.0 — Start here — the evolution of non-deterministic threat simulation | gets a red team that attacks the pipeline rather than the application |
| The findings it produces | E2.5 — Rules generated from an incident — and the benign corpus that decides them | become detections, and the benign corpus that decides them |
| The attestations from C2.18 | F1.5 — Evaluation output as audit evidence | become evaluation output offered as audit evidence |
| Pitfall | What to do | If you are running it for a room |
|---|---|---|
| Calling the model pass the scan | The deterministic pass is what the model pass is scored against. Run C2.3 both ways and compare — the lesson prints both. | — |
| Reading a smaller queue as a better queue | C2.4 and C2.5 cut findings for two different reasons — duplicates, and unreachable code — and only one of them is about risk. A queue cut by the wrong filter is a queue missing defects. | — |
| Treating the pentest lessons as an attack class | C2.14 is in the chapter because an offensive agent needs controls around it. Do not skip it to get to C2.19. | Say before C2.10 that C2.14 is coming, or the room will ask for it four lessons early. |
| Accepting a green CI run as proof a vulnerability is closed | C2.16 is the whole lesson: the test has to fail against the unpatched code. Hand it one that passes both ways and it refuses. | — |
| When | Prompt |
|---|---|
| Starting a lesson | Summarise lesson <id>'s Risk, Control and Day 0/1/2 in plain English, under 100 words. |
| See what changed | Run scripts/checkpoint.py --at <id> --diff and explain the changed lines like I'm new to Python. |
| Predict before "Your turn" | Before you run it: here is my prediction for the Your turn change: <prediction>. Now run it and tell me if I was right. |
| Stuck on an error | Show me the exact error, what it means, and the smallest fix. Don't change anything else. |
| Check understanding | Quiz me with 3 questions on lesson <id>. Don't give answers until I reply. |
| Connect to the attack side | Which Function B lesson attacks what I built in <id>, and how, in two sentences? |
| When something will not run | Run scripts/doctor.py and tell me only the lines that are not ok, with the fix for my operating system. |
| Term | In one line |
|---|---|
| Agent | Software that plans, calls tools and acts on what it reads |
| Harness | The loop around the model: context in, tool call out, result back |
| Verifier | Code, not a model, that checks a result before it is accepted |
| MCP | Model Context Protocol — a standard way for agents to reach tools |
| Sink | Somewhere data can do damage |
Audience (aud) | The one server a token is valid at |
| Scope | The one permission a token grants |
| Workload identity | A name for a piece of software, e.g. spiffe://…/agent/workflow |
| Token exchange | Swapping a user token plus an agent identity for a short, narrow, per-action token |
| A2A | Agent-to-agent messaging |
| Span / trace id | One recorded step of a run / the id that joins all of them |
| Canary token | A credential that looks real and that nothing legitimate ever uses |
| Blast radius | What one run can reach and damage before a human can stop it |
| Conformance vs. accuracy | Whether the answer has the right shape vs. whether it is right |