Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.
recordings/C2.4.mp4Parallel analysis tracks report the same bug three times, and some of those bugs do not exist.
Stages 8–9: consolidate overlapping findings, then cross-reference each one against syntax and imports to weed out hallucinations.
Deduplicate findings across three analysis tracks, then verify each against the AST and drop the ones that reference code that is not there.
The corpus is a write surface. Training data, a RAG index and an agent's memory are three versions of the same problem: text somebody else authored, read back later as fact, long after anyone remembers where it came from.
dedup() and cross_reference() in findings.py. Three tracks report search_bookings and the queue gets one row that remembers all three bases — a defect two unrelated tracks reached is a better bet than one. Then the cheapest check in the pipeline: a finding naming a function nobody wrote is refuted, and it is otherwise perfectly formed.Two cheap stages that decide whether the queue is usable: collapsing the reports of one defect into one row while keeping how many independent tracks reached it, and refuting the finding that names a function nobody wrote — which is otherwise perfectly formed.
Parallel analysis tracks report the same bug three times, and some of those bugs do not exist.
Consolidate overlapping findings, then cross-reference each against syntax and call graph.
Findings before and after consolidation, and how many survived verification. Both numbers, not just the smaller one.
Measure this
| what to measure | denominator or target |
|---|---|
| findings before and after consolidation | two counts, both reported |
| findings that survived verification | count |
three names for one problem: text somebody else wrote, read back as fact training data --+ RAG corpus --+--> context window --> the agent believes it agent memory --+ detection differs per layer; provenance is the control in all three
Stage 7 ran several analysers in parallel. That produces two problems this stage exists to solve, and they are different problems.
Stage 8 — Deduplication. Three analysers find the same bug and report it three times. Worse, they report it at slightly different line numbers with different CWE labels, so naive matching does not collapse them. An engineer who sees the same bug three times stops trusting the count.
Stage 9 — Contextual verification. Cross-reference each finding against the actual syntax and imports to weed out hallucinations. This is the cheapest, highest-yield filter in the whole pipeline, because a model finding that references a function that does not exist, or a module that was never imported, is provably wrong — no judgement required.
The order matters: deduplicate first, then verify, or you spend verification effort on three copies of the same claim.
Where you are in the pipeline. `` [Ingestion & Mapping] ──> [Threat Modelling] ──> [Discovery] └─ stages 1-4 └─ stages 5-6 └─ stages 7-10 ──> [Dynamic Validation] ──> [Reporting] └─ stages 11-14 └─ stage 15 ``Two findings are the same defect if they name the same sink in the same function, even at different lines and under different CWE labels. Cluster on that, and keep the best-evidenced member.
Now check each surviving claim against the code. Three checks, all mechanical, none requiring judgement.
Eight raw findings go in. Five distinct defects come out of that. Three survive to the end.
The queue and the file are both C2.3's: cybertravels/tools/bookings_api.py, read off disk. The CWE-89 is where Semgrep put it. The CWE-639 on get_booking is what the model pass guessed, and it arrives here to be checked rather than believed.
The skill does three things, in order:
Why step 3 earns its place. A finding about os.system in a file that never imports os should die here, not in a maintainer's inbox.
Now watch something survive that should not: a CWE-89 on list_my_bookings, whose query is parameterised and scoped to the session.
Neither stage rejects it, and neither is supposed to. These two stages remove what is provably duplicated and provably absent. Judging a claim about code that genuinely exists is triage, and triage is C2.15.
The skill — skills/appsec/finding-dedup-and-verification/SKILL.md
name: finding-dedup-and-verification
description: >-
Collapse raw findings from several tools into distinct defects across CWE
aliases and duplicate reports, then reject the ones whose symbols do not
appear in the file. Use when a pipeline produces more findings than defects,
or when a scanner reports something that is not in the code.
allowed-tools: Read, Grep, GlobA pipeline running three analyses over one file produces findings that overlap in two different ways — the same defect reported under an alias, and the same defect reported by three tools — and, if a model is one of the tools, findings about symbols that do not exist. Both have to be resolved before a human sees the list, or the human resolves them by ignoring the list.
Between analysis and triage, in any pipeline with more than one analyser, and always when a model is one of them.
1 — Normalise the CWE. Maintain an alias map — CWE-943 is CWE-89 for this purpose — and normalise before comparing. Aliases are why the same defect appears twice with different identifiers.
2 — Key each finding by defect, not by report. File, enclosing function, normalised CWE. Line numbers move; the enclosing function does not, and it is what makes two reports of one defect collapse.
3 — When duplicates disagree, keep the one with the best evidence. A taint result carries a path; a grep hit carries a line; a model result carries a sentence. Prefer in that order, and record which tools agreed — agreement is useful signal even though it is not proof.
4 — Verify each survivor against the source. Does the symbol it names appear in the file? Is the module it blames imported? A finding about os.system in a file that never imports os is a hallucination, and it is rejected here rather than in a maintainer's inbox.
5 — Report both counts and the rejects. Raw findings, distinct defects, and the rejected list with the reason. Discarding hallucinations silently loses the measurement of how often the analyser produces them.
Input — the queue at the top of scripts/finding_dedup_and_verification.py, verified against cybertravels/tools/bookings_api.py read off disk. The line numbers are real line numbers in that file: the CWE-89 is where Semgrep put it, and the CWE-639 is the hypothesis C2.3's model pass emitted on get_booking. Edit the queue and re-run — the buckets, counts and verdicts are derived from it, not hard-coded.
Output — the opening lines of a real run:
8 raw findings from 3 tracks, against tools/bookings_api.py
grep CWE-89 tools/bookings_api.py:41 execute conf=0.50
taint CWE-89 tools/bookings_api.py:41 execute conf=1.00
model CWE-89 tools/bookings_api.py:42 execute conf=0.93
model CWE-943 tools/bookings_api.py:41 execute conf=0.71
model CWE-639 tools/bookings_api.py:20 get_booking conf=0.82
model CWE-89 tools/bookings_api.py:47 execute conf=0.44
model CWE-798 tools/bookings_api.py:8 DB_PASSWORD conf=0.88
model CWE-78 tools/bookings_api.py:34 os.system conf=0.67Eight become five, then three. Two of the three are real, and the third — a CWE-89 on list_my_bookings, whose query is parameterised — is not. Neither stage rejects it, and neither is supposed to: these two remove what is provably duplicated and provably absent. Judging a claim about code that exists is triage, and it is a later stage.
The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"raw": 0,
"defects": [{"key": "str", "file": "str", "function": "str", "cwe": "str",
"reported_by": ["str"], "kept_from": "str"}],
"rejected": [{"finding": "str", "reason": "symbol absent|module not imported"}],
"counts": {"raw": 0, "distinct": 0, "rejected": 0}
}Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.
The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.
Install it, once. From the folder you cloned. --all does every agent it finds on your machine:
python3 scripts/install_skills.py --all --lessons C2.4Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.
Then pick c2-4-deduplication-and-contextual:
| agent | install just for this agent | start it |
|---|---|---|
| Claude Code | python3 scripts/install_skills.py --tool claude --lessons C2.4 | type /c2-4-deduplication-and-contextual |
| GitHub Copilot (VS Code) | python3 scripts/install_skills.py --tool copilot --lessons C2.4 | type /c2-4-deduplication-and-contextual in Chat |
| Cursor | python3 scripts/install_skills.py --tool cursor --lessons C2.4 | type / in Agent chat and search for it |
| Codex | python3 scripts/install_skills.py --tool agents --lessons C2.4 | type $c2-4-deduplication-and-contextual, or run /skills |
No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.
python3 scripts/lesson.py C2.4Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.
python3 scripts/checkpoint.py --at C2.4 --out work/cybertravels
python3 scripts/checkpoint.py --at C2.4 --diffOn Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.
Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.
Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.
If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.
Eight raw findings collapse to five distinct defects, with the CWE-943 alias merging into CWE-89 and the taint result kept over grep and model duplicates. Contextual verification then rejects the hallucinated DB_PASSWORD and os.system findings because neither symbol appears in the file, leaving three: the real SQL injection in search_bookings, C2.3's IDOR hypothesis on get_booking, and one false positive on list_my_bookings that no mechanical check can kill.
Add a fourth verification check: does the CWE class match the sink type? A CWE-22 finding on a conn.execute call is provably mislabelled, and that check costs nothing to run.
Tools used
Skill source: skills/appsec/finding-dedup-and-verification