Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.
recordings/A2.1.mp4A trace holding only successful calls hides exactly the events worth alerting on, and a token pasted into a span is a credential in the log pipeline.
One trace id joining the reasoning to the audit row; refusals as first-class spans.
Emit a run's spans and join them to the audit log by trace id.
The run produced the right answer, so nobody looked further. The trace would have shown that it reached for the refund tool first, was refused by the policy, and tried a different route — which is a signal about the input it was given, not about that run. Emitting only the answer threw away the only interesting part.
cybertravels/observability.py. Every span carries the trace id that joins it to the audit row, which is the join Function E's detections are written against.The agent run as spans — which agent, which step, which tool, what the delegated token said — with one trace id joining reasoning to action, secrets summarised rather than carried, and refusals recorded with the boundary that produced them.
A run that emits only its answer is unreviewable, and a trace of successes hides exactly the events worth alerting on.
Emit the run as spans, joined by a trace id, with tokens summarised and refusals recorded with the boundary that produced them.
Spans per run carrying the trace id: all of them. Credentials appearing in a span: zero, and worth grepping for rather than believing.
Measure this
| what to measure | denominator or target |
|---|---|
| spans per run carrying the trace id | share of spans; target 1.00 |
| credentials appearing in a span | count; target zero, and grep for it rather than believe it |
THE RUN, AS SPANS
thought ─ plan ─ approval ─ token_issued ─ tool_result ─ final
│ │
denied claims only,
at=policy never the token
at=human
at=resource ◄── WHERE it was refused is the useful partA run that emits only its final answer is unreviewable. What an operator needs is the shape of the run, and that is a trace.
CyberTravels emits eleven kinds by the end of this lesson, and the vocabulary is the design. Count them in cybertravels/observability.py — the list below is all of them, not a selection:
start the run opened, with the trace id everything else carries
thought what the model said between calls
plan the tool it chose, and the scope that implies
approval a human granted or refused
token_issued the delegated claims — summarised, never the token
denied and WHERE: policy, human, verifier, or resource server
tool_result what came back
budget a ceiling was reached
error something raised, and the run says so
final the answer
done the run closed, so a truncated trace is detectableThe four at the edges — start, error, final, done — are the ones that get left out of a hand-rolled tracer, and they are what make a missing span visible. A trace with no done is either a run still going or a run that died, and without the pair you cannot tell which.
denied carrying where it was refused is the one that repays itself. Four places refuse, and each is a different incident:
Collapse them into one denied with no at and all four read identically in the log, which is how "the agent was denied 90 times last week" becomes a sentence nobody can act on.
cybertravels/observability.py is the file. The skill below checks whether a recorded run can actually be replayed and reasoned about — which is a stronger property than "we have logs".
The skill — skills/response/run-replayability-audit/SKILL.md
name: run-replayability-audit
description: >-
Check whether an incident run can be replayed at all — model version, seed,
prompt, tool results, retrieved context — and what a later model version does
to the replay. Use when forensics needs to know why the agent did what it did.
allowed-tools: Read, Grep, Glob"Why did the agent do that" is answerable only if the run can be re-run under the conditions it ran in. That needs the model version, the sampling parameters and seed, the exact prompt, every tool result and the retrieved context. A typical production run records enough to see what happened and not enough to reproduce it.
Before an incident, as a readiness check, and during one, to establish honestly whether the reconstruction is possible.
1 — List the five inputs and check each against a real run record. Model version and seed are the two usually missing, and their absence is decisive rather than inconvenient.
2 — Attempt the replay. If any input is missing, say what the replay can and cannot establish. A partial replay is still useful for the tool path and useless for the reasoning.
3 — Replay under later model versions. The provider has probably upgraded. Record whether the action changes: if it does, the original decision cannot be reproduced at all, and that is a finding about the estate rather than about the incident.
4 — Cost full instrumentation. Storage and latency for recording everything, against the incidents where you needed it. Present both; the answer is usually to instrument the high-tier agents only, and that is a defensible decision when the numbers are attached.
5 — Record what the estate has chosen. Which agents are replayable and which are not, so nobody assumes during an incident.
Input — the fixture committed at the top of scripts/run_replayability_audit.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.
Output — the opening lines of a real run:
fully instrumented replayable=True
typical production replayable=False
✗ model version — a silent upgrade changes the output
✗ seed — sampling makes the run unrepeatable
prompts only replayable=False
✗ tool results — the agent saw a world you cannot rebuild
✗ model version — a silent upgrade changes the output
✗ seed — sampling makes the run unrepeatableThe run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"inputs": [{"name": "str", "recorded": false}],
"replay": {"possible": false, "establishes": ["str"], "cannot_establish": ["str"]},
"version_drift": [{"version": "str", "action": "str", "same_as_original": false}],
"cost": {"storage_per_run": "str", "latency_ms": 0},
"policy": [{"tier": "str", "fully_instrumented": true}]
}Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.
The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.
Install it, once. From the folder you cloned. --all does every agent it finds on your machine:
python3 scripts/install_skills.py --all --lessons A2.1Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.
Then pick a2-1-observability:
| agent | install just for this agent | start it |
|---|---|---|
| Claude Code | python3 scripts/install_skills.py --tool claude --lessons A2.1 | type /a2-1-observability |
| GitHub Copilot (VS Code) | python3 scripts/install_skills.py --tool copilot --lessons A2.1 | type /a2-1-observability in Chat |
| Cursor | python3 scripts/install_skills.py --tool cursor --lessons A2.1 | type / in Agent chat and search for it |
| Codex | python3 scripts/install_skills.py --tool agents --lessons A2.1 | type $a2-1-observability, or run /skills |
No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.
python3 scripts/lesson.py A2.1Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.
python3 scripts/checkpoint.py --at A2.1 --out work/cybertravels
python3 scripts/checkpoint.py --at A2.1 --diffOn Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.
Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.
Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.
If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.
A run's spans in order, each carrying the trace id, tokens present only as summarised claims, and every refusal appearing with the boundary that produced it.
Put the whole delegated token in a span instead of its claims. Nothing breaks, the trace is more useful, and you have just put a credential in your log pipeline. That trade is made by accident constantly.
Tools used
Skill source: skills/response/run-replayability-audit