Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.
recordings/E2.4.mp4Coverage gaps nobody mapped.
Detection-as-code with agents inside the CI loop.
Generate and unit-test Sigma rules in CI; map coverage to ATT&CK.
An agent can write and tune a detection far faster than you can, which means it can also ship a confident, wrong rule into production far faster than you can. The validation discipline is the whole of the value.
review() calls D1.6's deployable() rather than restating it. Your sequence rule ships at 1,000 events a day and needs eleven analysts at 100,000 — same rule, same precision. And a candidate with no technique mapped is refused, because a rule nobody mapped is a rule nobody can reason about for coverage.The gate between a model-written detection and a deployed one: a measured false-positive rate at the volume it will actually see, a technique mapping so coverage can be reasoned about, and a human who accepts it.
An agent writes candidate rules faster than anyone can review them, and a rule at 5% precision is not 5% useful — it is negative.
Let the loop generate, then score every candidate against real historical telemetry before anything ships.
Firing volume per true positive. A rule at 301 alerts to 1 is rejected with its number rather than with an opinion.
Measure this
| what to measure | denominator or target |
|---|---|
| firing volume per true positive | ratio; 301 to 1 is rejected with its number |
agent writes rule --> test corpus --> tuned rule --> production
^
+------+-------+
| true positives from history |
| benign traffic that must |
| NOT fire |
+-----------------------------+
the speed is real. so is the speed of shipping a wrong rule.Using an agent to write detections is genuinely effective: it produces candidate rules quickly, across more log sources than a human would attempt.
What it cannot supply is the judgement that decides whether a rule ships, because that judgement depends on a cost the telemetry does not contain: analyst trust. A rule with 5% precision is not 5% useful — it is negatively useful, because it spends attention that the good rules need.
So the workflow is: the agent generates candidates, and a scoring step against real historical telemetry decides which survive. The scoring step is the job, and it is the part teams skip.
Anchor → E1.0. This shortens the gap between a rule is needed and a rule exists, which is a real interval and not the one that matters most. What it cannot shorten is the decision to ship, because the cost that decides it — analyst trust — is not in the telemetry.
All five detect something. R1 has perfect recall on http traffic and would put 301 alerts a day in the queue. R4 has 100% precision on nothing useful. The deployable set is decided by a threshold nobody writes down.
Every candidate rule detects something. The skill replays each against real history and scores the third property nobody checks — firing volume — so a rule that produces 301 alerts for one true positive is rejected with its numbers rather than with an adjective.
The skill — skills/detection/detection-rule-deployability/SKILL.md
name: detection-rule-deployability
description: >-
Score candidate detection rules on precision, recall and firing volume, and
reject the ones no analyst could work regardless of how well they detect. Use
when authoring detections with or without a model, or when a rule is proposed
because it caught the incident.
allowed-tools: Read, Grep, GlobEvery candidate rule detects something. Deployability is a different property and it is arithmetic: precision, recall, and how many times the rule fires per day against real history. A rule failing on the third is rejected however good the first two look, because it will be muted within a week.
Authoring detections, reviewing a model's proposed rules, and any time a rule is proposed on the strength of catching one incident.
1 — Replay each candidate against real history. Not a sample chosen to contain the incident — the actual period, including the quiet parts.
2 — Compute precision, recall and volume. Volume is the one people omit and the one that decides whether the rule survives contact with an analyst.
3 — Set a deployability bar before you look at the results. Precision floor, recall floor, and a maximum firings per day. Setting it afterwards means setting it around the rule you like.
4 — Reject the broad rules explicitly, with their numbers. "Rejected: 301 firings for 1 true positive" is a sentence the author can act on; "too noisy" is not.
5 — Look at what survived, and what it depends on. A high-precision rule usually depends on a specific field being populated. Record that dependency — it is the thing that will silently break the rule later.
Input — the fixture committed at the top of scripts/detection_rule_deployability.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.
Output — the opening lines of a real run:
history: 522 events, 2 true positives
rule alerts prec recall alerts/TP
----------------------------------------------------------------------
R1 any http_get by an agent 301 0.003 0.500 301.0
R2 http_get to a non-github host 1 1.000 0.500 1.0
R3 link-local address 1 1.000 0.500 1.0
R4 any failed action 20 0.000 0.000 inf
R5 credential path OR link-local 2 1.000 1.000 1.0The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"history": {"events": 0, "period_days": 0, "true_positives": 0},
"candidates": [{"name": "str", "fires": 0, "tp": 0, "precision": 0.0, "recall": 0.0,
"per_day": 0.0, "verdict": "deploy|reject", "why": "str"}],
"bar": {"precision": 0.0, "recall": 0.0, "max_per_day": 0},
"dependencies": [{"rule": "str", "requires_field": "str"}]
}Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.
The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.
Install it, once. From the folder you cloned. --all does every agent it finds on your machine:
python3 scripts/install_skills.py --all --lessons E2.4Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.
Then pick e2-4-agent-assisted-detection-engineering:
| agent | install just for this agent | start it |
|---|---|---|
| Claude Code | python3 scripts/install_skills.py --tool claude --lessons E2.4 | type /e2-4-agent-assisted-detection-engineering |
| GitHub Copilot (VS Code) | python3 scripts/install_skills.py --tool copilot --lessons E2.4 | type /e2-4-agent-assisted-detection-engineering in Chat |
| Cursor | python3 scripts/install_skills.py --tool cursor --lessons E2.4 | type / in Agent chat and search for it |
| Codex | python3 scripts/install_skills.py --tool agents --lessons E2.4 | type $e2-4-agent-assisted-detection-engineering, or run /skills |
No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.
python3 scripts/lesson.py E2.4Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.
python3 scripts/checkpoint.py --at E2.4 --out work/cybertravels
python3 scripts/checkpoint.py --at E2.4 --diffOn Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.
Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.
Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.
If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.
All five rules detect something. R1 fires 301 times for 1 true positive; R5 fires twice for 2 true positives with perfect precision and recall. The deployability check rejects the broad rules and the failed-action rule, shipping only the precise ones with a small daily queue impact.
Set your own alerts-per-true-positive budget and apply it to the rules already in production. Most SOCs discover that several long-standing rules would not pass the bar they would set today.
Tools used
Skill source: skills/detection/detection-rule-deployability