The skill — skills/appsec/appsec-exploit-validate/SKILL.md

name: appsec-exploit-validate
description: >-
  Prove a candidate vulnerability is real by reproducing it in an isolated
  sandbox, then decide whether it chains into something worse. Use when asked
  to validate or confirm a finding, write or run a proof-of-concept, check
  exploitability, reproduce a CVE, or separate theoretical findings from
  demonstrated ones.
allowed-tools: Bash, Read, Write, Grep

AppSec pipeline · Phase 4 — Dynamic validation

Covers stages 11–14. A finding that has been reproduced is a different object from a finding that has been argued for. This phase produces the difference, and it is the phase with the sharpest safety constraints.

When to use this

Only on findings that reached feasible: true, verdict: confirmed in Phase 3. Validating an unfiltered list wastes the most expensive stage in the pipeline on findings Phase 3 would have deleted.

Safety preconditions — check before anything else

Refuse to proceed unless all hold, and say which one failed:

1. The target is a local sandbox or an explicitly authorised environment. Never a production host, never a third party's infrastructure.
2. The sandbox has no credentials beyond throwaway ones, and no route to a real network the exploit could reach.
3. The proof-of-concept demonstrates the flaw. It does not exfiltrate real data, persist, escalate beyond the demonstration, or damage anything.
4. There is a teardown and it runs even on failure.

This is not ceremony. A validation harness with production credentials in its environment is itself the vulnerability.

Procedure

Stage 11 — Sandbox replication. Stand the component up in isolation with the smallest fixture that reproduces the conditions: the vulnerable version, the reachable entry point, and nothing else. Record the exact commit and the fixture, because a PoC that cannot be re-run is an anecdote.

Stage 12 — Dynamic exploitation. Drive the entry point with an input that should trigger the sink. Capture the observable: the crash, the query executed, the file read, the process spawned. The observable is the evidence — an exit code is not. Record what you sent and what came back.

Stage 13 — Exploit chaining. Ask what the demonstrated primitive gives access to next. A path traversal that reads a config file containing a token is not a file-read bug; it is a credential-disclosure bug. Chain only within the sandbox, and stop at the first step that would need a real credential.

Stage 14 — Remediation engineering. Propose the fix at the right layer, and say what it costs. Prefer the control that makes the class impossible (parameterised queries, an allowlist, a type) over the one that blocks the sample payload (a regex on the input you happened to send). Then re-run stage 12 against the fix — a remediation that has not been tested against the PoC that motivated it is a hypothesis.

Example

Input — the fixture committed at the top of scripts/appsec_exploit_validate.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.

Output — the opening lines of a real run:

id     cwe       sev      requires                              grants
------------------------------------------------------------------------------------------------
F-01   CWE-200   low      ['unauthenticated']                   ['valid_report_id']
F-02   CWE-639   medium   ['authenticated', 'valid_report_id']  ['other_users_report_data']
F-03   CWE-352   medium   ['authenticated']                     ['password_reset_for_victim']
F-04   CWE-89    high     ['authenticated']                     ['database_read']
F-05   CWE-306   medium   ['internal_network']                  ['admin_actions']
F-06   CWE-918   medium   ['authenticated']                     ['internal_network']

The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.

Output contract

{
  "validations": [
    {"finding_id": "str", "reproduced": true,
     "sandbox": {"commit": "str", "fixture": "str", "isolated": true},
     "input": "str", "observable": "str",
     "chain": [{"primitive": "str", "leads_to": "str", "stopped_because": "str"}],
     "remediation": {"layer": "input|query|api|config|architecture",
                     "change": "str", "cost": "low|medium|high",
                     "retested": true, "still_reproduces": false}}
  ],
  "refused": [{"finding_id": "str", "precondition_failed": 1, "why": "str"}]
}

reproduced: false is a first-class result and must be reported, not dropped. A finding that survived Phase 3 and then failed to reproduce is the single most useful signal the pipeline produces about its own false-positive rate.

Failure modes

• Reporting reproduced: true without an observable. The observable is the claim; without it there is nothing to check.
• Treating "the exploit ran" as "the fix works". Re-run after remediation or set retested: false and say so.
• Chaining outside the sandbox. Stop and record stopped_because.
• Fixing the payload instead of the class. A regex that blocks ../ does not fix path traversal.

Handoff

Pass validations to appsec-triage-report. Findings that failed to reproduce keep their finding record and gain reproduced: false — the report must be able to distinguish "demonstrated" from "asserted".