Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.
recordings/E1.2.mp4A detection that worked last month is silently degraded.
Watch model updates, prompt changes, index refreshes, tool versions.
Change the model underneath and catch the detection regression.
Nothing was attacked. The model was upgraded, a prompt was edited, a tool changed its output format — and the behaviour of the system moved. Drift is the failure mode with no adversary, and it is far more common than the ones with one.
drift() reports which moved and, in its own column, which have no approver — the model version on the vendor's schedule, the retrieval corpus anyone can add a template to, and the MCP tool descriptions B1.9 says can be rewritten after you approved them.The surfaces that change an agent's behaviour without a commit — model version, system prompt, retrieval index, tool descriptions, memory — baselined and diffed, with the column that makes it a finding: which of them anybody approves.
A detection that worked last month is silently degraded, because the model, the prompt or the tool list changed and none of it was a code change.
Sign off a baseline, then watch the four surfaces that move without raising a ticket.
Time between a behaviour change and its detection — and the share of controls currently outside their freshness window.
Measure this
| what to measure | denominator or target |
|---|---|
| time between a behaviour change and its detection | minutes |
| controls outside their freshness window | share of controls |
nothing was attacked model upgraded ----+ prompt edited ----+---> behaviour moves ---> baselines stale tool changed ----+ detections silent drift is the failure mode with no adversary, and the common one the control: a fixed probe suite, run on every change
Drift monitoring answers what changes an agent's behaviour without going through the change management process built for code: a new model version, an edited prompt, an added tool. All of them invalidate the testing your controls were signed off against.
That is the precise claim: the control was tested against a behaviour that no longer exists. It has not failed; it is unevidenced, which is a different and more honest state.
Two things are needed:
F1.7 turns the second into a compliance posture. This lesson produces the signal.
Anchor → E1.0. Drift lengthens every interval at once, silently. The model was upgraded, a prompt was edited, and the control is now unevidenced against behaviour that no longer exists — so the discover clock restarts without anybody starting it.
| change surface | in change management? | what happens today |
|---|---|---|
| application code | yes | pull request, review, CI |
| agent prompt | no | edited in a console |
| tool manifest | no | a config change, with no threat-model diff |
| model version | no | provider-side; you may not be told |
| policy | yes | if it is in git — often it is not |
| approval settings | no | a toggle in an admin UI |
Six things can change how an agent behaves. Four of them never go near change management. The skill counts those four first.
Then it tracks the drift across a quarter and splits the rise in two:
The second part is the interesting one, because nothing in your change log explains it.
The skill — skills/detection/behavioural-drift-monitor/SKILL.md
name: behavioural-drift-monitor
description: >-
Track an agent's drift from its signed-off baseline across a quarter and count
how many of the surfaces that change its behaviour bypass change management.
Use when an agent behaves differently than it was approved to, with no code
change to point at.
allowed-tools: Read, Grep, GlobThe model version, the prompt, the tool manifest and the approval settings all change what an agent does, and none of them generates a change record. So drift is the only signal that the approved system and the running system have diverged, and it has to be measured rather than inferred from a changelog.
Continuously, for any agent under governance — and specifically after a provider announces a model update.
1 — Enumerate the change surfaces. Code, model version, system prompt, tool manifest, approval settings, retrieval corpus. For each, record whether a change produces a record anybody reviews.
2 — Count the ones that bypass change management. Four of six is typical. That count is the argument for measuring drift at all.
3 — Fix a baseline at sign-off. Tools, resources, rate, refusal rate — and the model version and prompt hash alongside, so a later drift can be attributed.
4 — Compute drift per period and attribute each rise. A jump that coincides with a model upgrade is a different conversation from a jump with no corresponding event, and the second is the one to escalate.
5 — Set a tolerance and a consequence. Beyond tolerance: re-attest, or revert. A drift figure with no consequence is a chart.
Input — the fixture committed at the top of scripts/behavioural_drift_monitor.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.
Output — the opening lines of a real run:
when event drift new tools
------------------------------------------------------------------
90d control signed off 0.000 []
60d prompt edited 0.100 []
30d tool added (no PR) 0.330 ['run_shell']
5d model upgraded by vendor 0.550 ['run_shell']
observed drift rate 0.00647 TVD/day
tolerance 0.25The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"surfaces": [{"name": "str", "produces_change_record": false}],
"bypass_count": 0,
"baseline": {"at": "str", "tools": ["str"], "model_version": "str", "prompt_hash": "str"},
"timeline": [{"period": "str", "drift": 0.0, "new": ["str"], "attributed_to": "str|null"}],
"tolerance": 0.0,
"consequence": "str"
}Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.
The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.
Install it, once. From the folder you cloned. --all does every agent it finds on your machine:
python3 scripts/install_skills.py --all --lessons E1.2Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.
Then pick e1-2-drift-monitoring:
| agent | install just for this agent | start it |
|---|---|---|
| Claude Code | python3 scripts/install_skills.py --tool claude --lessons E1.2 | type /e1-2-drift-monitoring |
| GitHub Copilot (VS Code) | python3 scripts/install_skills.py --tool copilot --lessons E1.2 | type /e1-2-drift-monitoring in Chat |
| Cursor | python3 scripts/install_skills.py --tool cursor --lessons E1.2 | type / in Agent chat and search for it |
| Codex | python3 scripts/install_skills.py --tool agents --lessons E1.2 | type $e1-2-drift-monitoring, or run /skills |
No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.
python3 scripts/lesson.py E1.2Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.
python3 scripts/checkpoint.py --at E1.2 --out work/cybertravels
python3 scripts/checkpoint.py --at E1.2 --diffOn Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.
Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.
Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.
If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.
Drift rises across the quarter from 0.0 at sign-off to roughly 0.35 after the model upgrade, with run_shell appearing as a new tool. Four of six change surfaces bypass change management. The observed drift rate yields a freshness window, and the 90-day-old control test is reported STALE rather than passing.
Compute the drift rate for one production agent from three months of telemetry, and set its control freshness window from that number rather than from the audit calendar.
Tools used
Skill source: skills/detection/behavioural-drift-monitor