Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.
recordings/E3.6.mp4The agent anchors on its first hypothesis and spends the whole incident gathering evidence for it.
An explicit plan record with a replan trigger, so an abandoned branch is visible in the trace rather than silently dropped.
Feed contradicting evidence mid-investigation and check the plan actually changes, and that the abandoned branch is recorded.
An agent forms a hypothesis at step one and spends the rest of the incident finding evidence for it. The evidence that should have stopped it supports nothing at all — which is exactly why an agent scoring only support reads it as noise and carries on.
Plan records the hypothesis, counts contradicting evidence, and makes replanning a trigger rather than a judgement call. The abandoned branch stays in the trace, because a branch that is not in the trace looks like one nobody considered.An investigation that is structurally allowed to change its mind: a plan record, a replan trigger that counts contradicting evidence, and abandoned branches that stay visible in the trace.
An agent scoring evidence only on support confirms its first theory and never terminates.
Force a replan when evidence refutes, and keep the abandoned branch visible in the trace.
Branches considered and dropped, visible to a reviewer. A trace with one branch is not an investigation.
Measure this
| what to measure | denominator or target |
|---|---|
| branches considered and dropped, visible to a reviewer | count; a trace with one branch is not an investigation |
evidence supports? the plan
1 refund at 03:14 misuse|theft agent-misuse
2 agent's session misuse agent-misuse
3 SPIFFE, no human misuse agent-misuse
4 plan has no --- nothing -- agent-misuse <- refutes, supports
refund step nothing, and an
agent scoring only
support skips it
5 vendor tool desc injection REPLAN -------> indirect-injection
6 desc changed injection indirect-injection
the abandoned branch stays in the trace: a reviewer must see that
agent-misuse was considered and dropped, not that it was never raisedEvery investigator is wrong at step one. What separates an investigator from an expensive autocomplete is what happens when the evidence stops fitting.
The way agents fail here is specific. They score evidence on whether it supports a hypothesis — so evidence that supports nothing reads as noise, when refutation is exactly what should force the replan. The agent keeps gathering support for a theory the evidence already killed.
Anchor → E1.0. An agent that scores evidence only on support never ends the interval; it spends the whole incident confirming step one. Refutation is what makes the investigation terminate, and the abandoned branch stays in the trace so a reviewer can see it did.
A reviewer needs to see that a hypothesis was considered and dropped, not that it was never raised. A conclusion with no visible alternatives looks inevitable, and an investigation that looks inevitable is one nobody can audit.
In CyberTravels the branch that gets abandoned is the obvious one: the workflow agent issued the refund, so the workflow agent did it. The evidence that kills that theory supports nothing at all — the agent's own plan for that run contains no refund step.
Six pieces of evidence, three hypotheses, one replan. Step 4 is the one an eager investigator skips.
The skill — skills/secops/investigation-replan-trace/SKILL.md
name: investigation-replan-trace
description: >-
Run an investigation that must abandon its opening hypothesis, and check the
replan is visible in the trace. Use when an investigating agent keeps
confirming its first theory, when evidence that refutes rather than supports
is being ignored, or when a reviewer needs to see which branches were dropped.
allowed-tools: Read, Grep, GlobEvery investigator is wrong at step one. What separates an investigator from an expensive autocomplete is what happens when the evidence stops fitting: a real one abandons the branch, and the abandoned branch stays in the record.
The specific way agents fail here is subtle. They score evidence on whether it supports a hypothesis, so evidence that supports nothing reads as noise — when refutation is exactly what should force the replan.
On any investigation an agent drives end to end, and in review of one that reached a confident conclusion quickly. A trace with no replan on a complex incident is a warning, not a success.
1 — Record the opening hypothesis explicitly. An unstated hypothesis cannot be abandoned, only drifted from.
2 — Score each piece of evidence for and against, separately. Support and refutation are not one axis.
3 — Trigger a replan on refutation, not on absence of support. Evidence that supports nothing is the signal; treat it as a trigger, not as noise.
4 — Keep the abandoned branch in the trace. A reviewer must see that a hypothesis was considered and dropped, not that it was never raised.
5 — Stop when a hypothesis survives evidence that could have refuted it. Not when one accumulates the most support.
Input — six pieces of evidence and three competing hypotheses, in scripts/investigation_replan_trace.py.
Output — the turn of a real run:
4. traces the agent's plan for that run contains no refund step
(neutral — refutes nothing, so nothing changes)
5. mcp the vendor MCP server returned a tool description naming a refund
REPLAN agent-misuse -> indirect-injection{
"steps": [{"n": 0, "source": "str", "fact": "str",
"hypothesis_before": "str", "hypothesis_after": "str",
"replanned": true}],
"replans": 0,
"final": "str"
}Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.
The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.
Install it, once. From the folder you cloned. --all does every agent it finds on your machine:
python3 scripts/install_skills.py --all --lessons E3.6Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.
Then pick e3-6-plan:
| agent | install just for this agent | start it |
|---|---|---|
| Claude Code | python3 scripts/install_skills.py --tool claude --lessons E3.6 | type /e3-6-plan |
| GitHub Copilot (VS Code) | python3 scripts/install_skills.py --tool copilot --lessons E3.6 | type /e3-6-plan in Chat |
| Cursor | python3 scripts/install_skills.py --tool cursor --lessons E3.6 | type / in Agent chat and search for it |
| Codex | python3 scripts/install_skills.py --tool agents --lessons E3.6 | type $e3-6-plan, or run /skills |
No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.
python3 scripts/lesson.py E3.6Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.
python3 scripts/checkpoint.py --at E3.6 --out work/cybertravels
python3 scripts/checkpoint.py --at E3.6 --diffOn Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.
Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.
Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.
If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.
One replan, from agent-misuse to indirect-injection, with the neutral evidence at step 4 marked as refuting nothing — and the abandoned branch still visible in the trace.
Add evidence that refutes the final hypothesis too. An investigation that cannot end undecided is not investigating.
Tools used
Skill source: skills/secops/investigation-replan-trace