Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.
recordings/E1.3.mp4Prompts, traces, tool calls and approvals never reach the SIEM.
Onboard agent telemetry deliberately; decide retention.
Ship OTEL agent traces into OpenSearch and query them.
The agent holds a person's authority and acts in their name, so conventional UEBA reads it as that person behaving strangely at 03:00. Find it on behaviour instead, and you inherit its trace — which contains the reasoning, and whatever was in the context window when it ran.
retention_plan() decides per field rather than per record, because a record-level rule is decided by its most sensitive field: the whole run gets the prompt's retention and the investigation loses the chain. Here prompt_text is seven days and chain, trace_id and motive_origin are a year, which is the set E3.4 needs and the smallest one.Onboarding what the agent already emits, and settling the retention argument at the granularity where it can be won: per field, so the parts that make a run attributable outlive the parts that are somebody's prose.
An agent nobody registered acts under a person's credential, and conventional analytics read it as that person behaving oddly.
Score actors on regularity, rate and continuity, then decide per-field retention on the traces you inherit.
Unregistered actors surfaced, and the threshold chosen by expected cost — a flagged human costs half an analyst-hour, a missed agent costs forty.
Measure this
| what to measure | denominator or target |
|---|---|
| unregistered actors surfaced | count |
| threshold chosen by expected cost | the threshold — a flagged human costs half an analyst-hour, a missed agent forty |
1 · WHO IS ACTING? 2 · WHAT YOU THEN HOLD
the log says +----------------------------+
+------------------+ | prompts tool calls |
| user: dana@corp | | decisions identities |
| action: deploy | +----------------------------+
+------------------+ |
| none of it is in an
UEBA: "dana, strangely" application log, and the
| prompts carry whatever was
the missing field is not in the context window
"suspicious" - it is
"actor_type", recoverable v
from behaviour alone: retention is decided per
regularity, rate, continuity FIELD, not per record
find the actor first. its trace is what you then have to keep.The two questions in this lesson are one question asked twice: which of the actors in your logs is software, and what does its trace contain once you keep it? Neither has an answer in a standard SIEM, and the second only becomes urgent once the first one works.
The agents you most need to find are the ones in no registry (B3.7), and they act under a person's authority in a person's name — so conventional UEBA reads them as that person behaving strangely. Three behavioural signals separate them, none sufficient alone:
The error directions are not symmetric, and that is what sets the threshold. A human misclassified as an agent triggers an investigation: mild, and self-correcting. An agent misclassified as human stays invisible, which is the entire risk you were trying to address. Cost-weighting therefore picks a lower threshold than accuracy-maximisation would.
Once you have found it, agent telemetry turns out to have a property no other log source has: it contains the reasoning, not just the action. The trace records what the agent was trying to do, what it considered, and what the verifier said.
That is enormously useful for investigation and it is a retention and privacy problem, because reasoning traces contain whatever was in the context window — routinely customer data, source code and secrets the agent read legitimately. So retention is decided per field, not per record:
| Field | Forensic value | Sensitivity |
|---|---|---|
| timestamps, tool, target | high | low |
| verifier detail | high | low |
| acting identity + chain | high | low |
| model prompts | medium | high |
| tool results | high | high |
The first three are cheap and should be kept long. The last two are where the retention conversation actually is.
Anchor → E1.0. Both halves set the floor for all five intervals. An agent nobody registered has no clock running but its own behaviour, and on data that was never emitted discover does not take a long time — it never completes. The retention decision is the same argument backwards: it fixes how far into the past an investigation is allowed to reach.
The run record holds something that looks like a payment card number. It is in a source file the agent was entirely allowed to read.
The skill scans every field, then sets how long each field is kept separately:
Per field, not per record. Deleting the whole record to lose the card number also loses the evidence.
The run record contains a payment-card pattern, in a source file the agent read legitimately. The skill scans every field, then sets retention per field so timestamps and verdicts survive for 400 days and prompts do not survive 30.
The skill — skills/detection/agent-versus-human-scoring/SKILL.md
name: agent-versus-human-scoring
description: >-
Score actors on behavioural signals to separate agents from people, sweep the
threshold, and pick it by expected cost rather than by accuracy. Use when
deciding whether a session is automated, or when unregistered automation needs
finding.
allowed-tools: Read, Grep, GlobSeparating agent from human is a scoring problem with two asymmetric errors: a flagged human costs an analyst half an hour, and a missed agent costs whatever an unmonitored automation does. Choosing the threshold by accuracy weights those equally, which is the one thing you know is wrong.
Finding unregistered automation, deciding whether a session is a person, and before any control that treats agents differently from users.
1 — Score on behaviour, not on the user agent string. Inter-action variance, rate, breadth, and the share of actions with no preceding read. Anything self-declared is a claim.
2 — Score a spread of real actors. A service indexer, an unknown token, a person, a person driving an IDE assistant, and an agent deliberately jittered to look human. The last two are the interesting middle.
3 — Sweep the threshold and record both errors. Humans flagged and agents missed, at each setting. They move in opposite directions and the crossing point is not the answer.
4 — Attach a cost to each error and minimise the total. Analyst hours for a false positive, expected hours of an unmonitored agent for a false negative. The chosen threshold now has a justification somebody can argue with.
5 — Join to the registry. An actor scoring as an agent and absent from the registry is the finding worth routing; a registered agent scoring as an agent is working correctly.
Input — the fixture committed at the top of scripts/agent_versus_human_scoring.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.
Output — the opening lines of a real run:
actor score cv rate/s span_h truth
--------------------------------------------------------------
svc-indexer 0.800 0.0 20.04 0.01 agent
dana@corp 0.028 1.55 0.0 1.11 human
unknown-token-7f3c 0.563 0.0 1.0 0.11 agent
sam@corp-ide 0.533 0.0 0.5 0.1 human
polite-agent 0.021 1.26 0.0 0.83 agent
threshold humans flagged agents MISSEDThe run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"actors": [{"name": "str", "score": 0.0, "truth": "agent|human|unknown"}],
"sweep": [{"threshold": 0.0, "humans_flagged": 0, "agents_missed": 0, "expected_cost": 0.0}],
"costs": {"false_positive_hours": 0.0, "false_negative_hours": 0.0},
"chosen": {"threshold": 0.0, "why": "str"},
"registry": {"scored_agent_unregistered": ["str"]}
}The skill — skills/detection/agent-telemetry-retention/SKILL.md
name: agent-telemetry-retention
description: >-
Scan an agent's run record for sensitive content it read legitimately, then
set retention per field rather than per record so the investigable parts
survive and the prompts do not. Use when agent telemetry is being kept,
discarded, or argued about with privacy.
allowed-tools: Read, Grep, GlobAn agent run record is the richest telemetry in the estate and the most dangerous to keep whole. The content it read — source files, tickets, documents — lands in the record, so the record inherits every classification the source had. Per-record retention forces a choice between losing the investigation and keeping the data; per-field does not.
Before agent telemetry is retained at scale, and whenever a retention policy is being written by someone who has not read a run record.
1 — Read one full record. Every step, every field. This is the step people skip, and it is where the payment-card pattern in a legitimately-read source file turns up.
2 — Scan for sensitive patterns across all fields. Card numbers, keys, personal data, health terms. Record which field carried each hit — prompts and tool results are the usual answer.
3 — Classify fields by investigative value. Timestamps, tool name, target and verifier verdict answer most investigation questions. Prompts and raw tool output answer few and carry most of the risk.
4 — Set retention per field. Long for the structural fields, short for the content ones. Then age a record and check what an investigation could still do with it — that check is what makes the policy defensible.
5 — State what is lost. A short prompt retention means you cannot re-derive motivation after that window. Say so, rather than discovering it in an incident.
Input — the fixture committed at the top of scripts/agent_telemetry_retention.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.
Output — the opening lines of a real run:
full record (everything the harness saw):
{'n': 1, 'tool': 'read_file', 'target': '/work/repo/billing.py', 'verifier': 'n/a', 'ok': True, 'prompt': 'Investigate finding SEC-4471 in billing.py', 'result': 'def charge(card_number, amount): # card_number = 4111111111111111'}
{'n': 2, 'tool': 'search_code', 'target': 'charge(', 'verifier': 'n/a', 'ok': True, 'prompt': 'find callers of charge()', 'result': 'api/checkout.py:88 charge(user.card, total)'}
{'n': 3, 'tool': 'write_file', 'target': '/work/repo/billing.py', 'verifier': 'tests pass', 'ok': True, 'prompt': 'apply the fix', 'result': 'patch applied'}
sensitive content found in the trace:
step 1 result payment card
Nobody put a card number in the trace deliberately. The agent read aThe run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"record": {"steps": 0, "fields": ["str"]},
"scan": [{"pattern": "str", "field": "str", "legitimate_source": "str"}],
"retention": [{"field": "str", "days": 0, "investigative_value": "high|low"}],
"aged_record": {"age_days": 0, "questions_still_answerable": ["str"], "lost": ["str"]}
}Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.
The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.
Install it, once. From the folder you cloned. --all does every agent it finds on your machine:
python3 scripts/install_skills.py --all --lessons E1.3Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.
Then pick e1-3-bonus:
| agent | install just for this agent | start it |
|---|---|---|
| Claude Code | python3 scripts/install_skills.py --tool claude --lessons E1.3 | type /e1-3-bonus |
| GitHub Copilot (VS Code) | python3 scripts/install_skills.py --tool copilot --lessons E1.3 | type /e1-3-bonus in Chat |
| Cursor | python3 scripts/install_skills.py --tool cursor --lessons E1.3 | type / in Agent chat and search for it |
| Codex | python3 scripts/install_skills.py --tool agents --lessons E1.3 | type $e1-3-bonus, or run /skills |
No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.
python3 scripts/lesson.py E1.3Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.
python3 scripts/checkpoint.py --at E1.3 --out work/cybertravels
python3 scripts/checkpoint.py --at E1.3 --diffOn Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.
Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.
Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.
If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.
First: the service indexer and unknown token score highest, the human lowest, with the IDE user and the politely-jittered agent in between. The threshold sweep shows humans flagged rising and agents missed falling as it drops, cost-weighting selects a low one, and joining against the registry names the unregistered actors as shadow agents. Then the trace of one of them: a payment-card pattern in a source file it read legitimately, and per-field retention that keeps timestamps, tool, target and verifier for 400 days while dropping prompts at 30 and tool results at 7. After 90 days no sensitive content remains and the record still answers what the agent did.
Run the scoring against a week of your own authentication logs and count the actors it flags that are not in your registry. Then check the retention period on whatever traces you keep for them: if it matches your firewall logs, one of those two numbers was chosen without anyone looking at what the traces contain.
What you can do now. You can measure the discover interval instead of assuming it: four sensor classes scored against what an agent actually does, drift caught without a code change, and — as a bonus — the agents nobody registered found on behaviour, with their traces kept per field.
What you still cannot do. Four of the nine ordinary agent actions are seen by nothing you own, and the source that would see them lands nowhere. Everything here is a finding in a notebook; the detect interval is exactly where it was.
Chapter E2 builds the place it lands and the rules that read it: the lake, tiered by the queries the SOC runs, and detections mapped to ATT&CK and ATLAS.
Tools used
Skill source: skills/detection/agent-versus-human-scoring