Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.
recordings/D1.4.mp4Score actors on behavioural signals to separate agents from people, and pick the threshold by expected cost rather than accuracy.
To catch the actor you just played you have to see it, and an agent's tool calls arrive through a gateway that logs none of the fields telling an agent from a person. This is where the offensive finding becomes defensive telemetry.
redteam/actor.py reads the spans A2.1 already emits and scores four shape signals — rate, gap variance, tool breadth, burst. Your planner run scores above 0.7; a person's session scores below 0.4. Then the part that matters: pick_threshold() chooses by the cost of each mistake rather than by accuracy, and prints what accuracy would have picked. When those differ, accuracy was optimising something nobody asked for.Separating an agent from a person using only the shape of the trace — rate, gap variance, tool breadth, burst — and choosing the threshold by the cost of each mistake rather than by accuracy, since the two mistakes are never equally expensive.
To catch the actor you have just played, you have to be able to see it — and the model gateway logs none of the fields that would distinguish an agent from a person.
Wrap the gateway trace with identity, tool and arguments, then score actors on behaviour rather than on what they claim to be.
Threshold chosen by expected cost rather than accuracy, and the unregistered actors it surfaces.
Measure this
| what to measure | denominator or target |
|---|---|
| threshold chosen by expected cost rather than accuracy | the threshold, with the cost model behind it |
| unregistered actors surfaced | count |
the model gateway, by default wrapped for telemetry
{ } no per-call identity -> {"id": "svc-idx", "tool": "...",
{ } no tool arguments "args": {...}, "ts": ...}
{ } no acting principal |
v
score actors on behaviour:
regularity, rate, continuity
threshold set by COST, not accuracyThe pivot of the whole function: an offensive finding becomes defensive telemetry. To catch the actor you have just played, you have to be able to see it at all — and an agent's tool calls arrive through a model gateway that, by default, logs none of the fields that would tell an agent apart from a person.
Establishing telemetry means JSON-wrapping the gateway trace — prompt, tool, arguments, identity — and then scoring actors on behaviour, because the ones you most need to find are the ones in no registry, acting under a human's credential at a machine's tempo.
The skill looks at five CyberTravels accounts and works out which are people and which are software. It judges them on how they behave, not on what they call themselves — anyone can set a label.
Then it has to choose a cut-off. It chooses by cost, not by accuracy:
Those numbers are eighty times apart. So the cut-off with the best accuracy score is not the cut-off you want.
The skill — skills/detection/agent-versus-human-scoring/SKILL.md
name: agent-versus-human-scoring
description: >-
Score actors on behavioural signals to separate agents from people, sweep the
threshold, and pick it by expected cost rather than by accuracy. Use when
deciding whether a session is automated, or when unregistered automation needs
finding.
allowed-tools: Read, Grep, GlobSeparating agent from human is a scoring problem with two asymmetric errors: a flagged human costs an analyst half an hour, and a missed agent costs whatever an unmonitored automation does. Choosing the threshold by accuracy weights those equally, which is the one thing you know is wrong.
Finding unregistered automation, deciding whether a session is a person, and before any control that treats agents differently from users.
1 — Score on behaviour, not on the user agent string. Inter-action variance, rate, breadth, and the share of actions with no preceding read. Anything self-declared is a claim.
2 — Score a spread of real actors. A service indexer, an unknown token, a person, a person driving an IDE assistant, and an agent deliberately jittered to look human. The last two are the interesting middle.
3 — Sweep the threshold and record both errors. Humans flagged and agents missed, at each setting. They move in opposite directions and the crossing point is not the answer.
4 — Attach a cost to each error and minimise the total. Analyst hours for a false positive, expected hours of an unmonitored agent for a false negative. The chosen threshold now has a justification somebody can argue with.
5 — Join to the registry. An actor scoring as an agent and absent from the registry is the finding worth routing; a registered agent scoring as an agent is working correctly.
Input — the fixture committed at the top of scripts/agent_versus_human_scoring.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.
Output — the opening lines of a real run:
actor score cv rate/s span_h truth
--------------------------------------------------------------
svc-indexer 0.800 0.0 20.04 0.01 agent
dana@corp 0.028 1.55 0.0 1.11 human
unknown-token-7f3c 0.563 0.0 1.0 0.11 agent
sam@corp-ide 0.533 0.0 0.5 0.1 human
polite-agent 0.021 1.26 0.0 0.83 agent
threshold humans flagged agents MISSEDThe run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"actors": [{"name": "str", "score": 0.0, "truth": "agent|human|unknown"}],
"sweep": [{"threshold": 0.0, "humans_flagged": 0, "agents_missed": 0, "expected_cost": 0.0}],
"costs": {"false_positive_hours": 0.0, "false_negative_hours": 0.0},
"chosen": {"threshold": 0.0, "why": "str"},
"registry": {"scored_agent_unregistered": ["str"]}
}Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.
The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.
Install it, once. From the folder you cloned. --all does every agent it finds on your machine:
python3 scripts/install_skills.py --all --lessons D1.4Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.
Then pick d1-4-establishing-telemetry-and-detecting-the:
| agent | install just for this agent | start it |
|---|---|---|
| Claude Code | python3 scripts/install_skills.py --tool claude --lessons D1.4 | type /d1-4-establishing-telemetry-and-detecting-the |
| GitHub Copilot (VS Code) | python3 scripts/install_skills.py --tool copilot --lessons D1.4 | type /d1-4-establishing-telemetry-and-detecting-the in Chat |
| Cursor | python3 scripts/install_skills.py --tool cursor --lessons D1.4 | type / in Agent chat and search for it |
| Codex | python3 scripts/install_skills.py --tool agents --lessons D1.4 | type $d1-4-establishing-telemetry-and-detecting-the, or run /skills |
No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.
python3 scripts/lesson.py D1.4Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.
python3 scripts/checkpoint.py --at D1.4 --out work/cybertravels
python3 scripts/checkpoint.py --at D1.4 --diffOn Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.
Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.
Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.
If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.
The service accounts and unregistered token score highest, the human lowest, and cost-weighting selects a low threshold that finds the shadow agents at the price of a few analyst-hours.
Check whether your model gateway logs the acting identity per call. If it logs only the API key, every agent is anonymous and this scoring is the only actor you have.
Tools used
Skill source: skills/detection/agent-versus-human-scoring