All lessons › Function E — The Agentic SOC › Detect — the Lake, and Rules Mapped to MITRE

E2.5

Rules generated from an incident — and the benign corpus that decides them

Maps toOWASP AgenticT8 MITRE ATLASDiscovery MITRE ATLASCollection NIST AI RMFMEASURE EU AI ActArt. 12

▶

Lightboard recording

Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.

Video placeholderpublishes automatically from recordings/E2.5.mp4

Risk

A rule generated from one incident matches that incident and nothing else, or matches everything and buries the queue.

Control

Generate, then measure against a benign corpus. A rule with no measured false-positive rate is not a rule, it is a guess.

The lab

Generate a rule from a trace, then score it against benign traffic and report the false-positive rate before deployment.

↗ Open this skill in the repository

◉Use case relevance

You have just reconstructed an incident and the rule almost writes itself. That is the problem: every rule you could write catches the incident, because you wrote it from the incident. What decides whether it ships is the traffic it fires on when nothing is wrong.

At CyberTravels. A rule generated from an incident matches that incident; that is not evidence. measure() runs it against the trace it came from and against a benign corpus, and reports both failure modes — overfitted to one trace_id, or general enough to match the baseline. A rule with no measured false-positive rate is a guess.

◴What this lesson is — Day 0, Day 1, Day 2

Why a rule generated from an incident always matches that incident, and the benign corpus that tells an overfitted rule from a generalising one — the measurement that turns a guess into a rule.

Day 0 — why

Every candidate rule catches the incident it was generated from, so catching it cannot be the test.

Day 1 — how

Score candidates against a benign corpus containing the hard cases — for a refund rule, legitimate refunds.

Day 2 — measure

False-positive rate on benign traffic: 21% buries the queue, 0% with generalisation ships, 0% matching only this incident is useless.

Measure this

what to measure	denominator or target
false-positive rate on benign traffic	21% buries the queue, 0% with generalisation ships, 0% matching only this incident is useless

▦The framework, and how it works

   incident trace
        |
        v
   +----------------------+     every candidate catches the incident,
   | candidate rules      |     so that cannot be the selection test
   |   any refund         |
   |   refund w/o approval|
   |   refund on BK-772   |
   +----------+-----------+
              |
              v
       benign corpus  (must contain LEGITIMATE refunds)
              |
      +-------+--------+---------------+
      v                v               v
   21% FP           0% FP           0% FP
   buries the       generalises     matches this
   queue            AND quiet       incident only
   REJECT           SHIP            REJECT

The fastest source of a good detection is an incident you have just had, and the trap is trusting it for that reason: a rule tests clean against the case it was built from, so catching that one case is not the property you can select on.

What separates a rule worth deploying is what it does to traffic that is not the incident. That means a benign corpus containing the hard cases: for a refund rule, legitimate refunds. A corpus of unrelated traffic proves nothing, because the naive rule looks clean against it.

Anchor → E1.0. The fastest route to a rule, and the one most likely to lengthen every other interval instead. A rule that fires on 21% of benign traffic spends the attention the rest of the queue needs, so the benign corpus is what decides, not the incident.

2 · Write the bad candidates on purpose

Two of these three rules are worth writing precisely because they will be rejected.

The naive generalisation takes the tool that appeared in the incident: any refund.
The over-fitted one takes the identifier: refund on BK-772.

Both of them catch the incident. One buries the alert queue. The other is worthless tomorrow, when the next booking has a different number.

Seeing them scored beside the good rule is the point. It turns the good rule into a choice you made rather than an assumption you started with.

3 · Generate, then measure before shipping

Three candidate rules. One corpus of benign traffic. One number each.

A rule with no measured false-positive rate is a guess with syntax.

▶Real time execution as skill

The skill — skills/detection/detection-rule-synthesis/SKILL.md

name: detection-rule-synthesis
description: >-
  Generate a detection rule from an incident trace and score it against benign
  traffic before it ships. Use after an incident is reconstructed, when turning
  a hunt finding into a standing detection, or when a proposed rule has no
  measured false-positive rate.
allowed-tools: Read, Grep, Glob

The incident tells you what to write; the benign corpus tells you whether to ship it

The fastest source of a good detection is an incident you have just had. The trap is that every candidate rule catches the incident — that is how it was generated — so catching it cannot be the property you select on.

What separates a rule worth deploying is what it does to traffic that is not the incident.

When to use this

Immediately after reconstruction, while the trace is still loaded, and again whenever a hunt hypothesis is promoted. Do not use it to write coverage from scratch: a rule generated from no incident has nothing to generalise from.

Step-by-step

1 — Take the reconstructed trace, not the alert. The alert is one event; the rule needs the sequence around it.

2 — Write more than one candidate, deliberately including a bad one. The naive generalisation ("the tool that appeared") and the over-fitted one ("this booking id") are both worth writing, because seeing them scored is the lesson.

3 — Assemble a benign corpus that contains the hard cases. Legitimate refunds, not just unrelated traffic. A corpus with no near-misses proves nothing.

4 — Score every candidate on the benign corpus and record the rate. Not a verdict, a number.

5 — Ship the one that generalises and stays quiet, and say why the others were rejected. The rejected candidates are the evidence that the shipped one was chosen rather than assumed.

Example

Input — a three-step incident and an 84-run benign corpus, in scripts/detection_rule_synthesis.py.

Output — a real run:

candidate rule                          fires on   FP  FP rate  verdict
any refund                                  True   18     21%  REJECT — buries the queue
refund with no approval in the run          True    0      0%  ship
refund on BK-772                            True    0      0%  REJECT — matches this incident only

All three catch the incident. Only one is a detection.

Output contract

{
  "benign_runs": 0,
  "candidates": [{"rule": "str", "catches_incident": true,
                  "false_positives": 0, "fp_rate": 0.0,
                  "verdict": "ship|REJECT"}]
}

Common edge cases

The over-fitted rule scores perfectly. Zero false positives and zero future value. Test for a literal identifier in the rule body.
The benign corpus is too easy. If it contains no legitimate refunds, the naive rule looks shippable.
The incident is the only positive. One true positive cannot establish recall; the rule's recall is unknown until it runs.

Failure modes

Selecting on "does it catch the incident". Every candidate does.
Shipping without a false-positive number. That is a guess with syntax.
Never writing the bad candidates. Then nothing shows the good one is good.

Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.

1In your AI agent — you are taught

The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.

Install it, once. From the folder you cloned. --all does every agent it finds on your machine:

python3 scripts/install_skills.py --all --lessons E2.5

Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.

Then pick e2-5-rules-generated-from-an-incident:

agent	install just for this agent	start it
Claude Code	`python3 scripts/install_skills.py --tool claude --lessons E2.5`	type `/e2-5-rules-generated-from-an-incident`
GitHub Copilot (VS Code)	`python3 scripts/install_skills.py --tool copilot --lessons E2.5`	type `/e2-5-rules-generated-from-an-incident` in Chat
Cursor	`python3 scripts/install_skills.py --tool cursor --lessons E2.5`	type `/` in Agent chat and search for it
Codex	`python3 scripts/install_skills.py --tool agents --lessons E2.5`	type `$e2-5-rules-generated-from-an-incident`, or run `/skills`

2As one Python command — you just run it

No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.

python3 scripts/lesson.py E2.5

3Read the code, and the difference — nothing runs

Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.

python3 scripts/checkpoint.py --at E2.5 --out work/cybertravels
python3 scripts/checkpoint.py --at E2.5 --diff

On Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.

Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.

Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.

If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.

✓What you just proved

All three candidates catch the incident; only the sequence rule ships. The naive rule fires on 18 legitimate refunds — a 21% false-positive rate on a business that runs on refunds.

✎Your turn

Add a benign run that the shipped rule fires on. If you cannot construct one, your corpus is too easy.

Tools used

Sigma

Skill source: skills/detection/detection-rule-synthesis