All lessons › Function A — Getting Started — Building Agentic AI › Harness Engineering — Making a Demo Into a System
A2.5

What you have built — and every way it can now go wrong

A2
▶
Lightboard recording

Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.

Video placeholderpublishes automatically from recordings/A2.5.mp4
Risk

Builders who never see their own system described adversarially ship the same defect in the next one.

Control

The same architecture map, annotated with what an attacker reaches for at each edge.

The lab

Compute the blast radius of the agent you just built.

◉Use case relevance

Everything in this system works. That sentence is true and it is the beginning of the next five functions rather than the end of this one — because 'works' was measured against what you intended, and nobody has yet measured it against somebody who intends otherwise.

At CyberTravels. The same CyberTravels architecture from A1.0, re-read by somebody who wants it to fail. Every row in that table is a lesson in Function B, by id.

◴What this lesson is — Day 0, Day 1, Day 2

The handover into Function B. Every component built in this function re-read as an attack surface, every control re-read as something with a bypass, and the agent's blast radius measured as the number the next function argues with.

Day 0 — why

A builder who has never seen their own system described adversarially ships the same defect in the next one.

Day 1 — how

Re-read every component you built as an attack surface, and measure what one run can reach and damage.

Day 2 — measure

Blast radius: objects reachable, the subset writable, and the irreversible actions among them. That number decides how much autonomy the agent can carry into Function B.

Measure this

what to measuredenominator or target
blast radiusobjects reachable, the subset writable, and the irreversible actions among them

▦The framework, and how it works

        SAME MAP, READ BY SOMEBODY ELSE

   ingress          ─►  prompt injection            B1.2
   vendor MCP       ─►  indirect injection          B1.3
   tool description ─►  rug-pull after approval     B1.9
   memory           ─►  persistence                 B1.4
   a2a              ─►  one becomes four            B1.7
   the human gate   ─►  saturation                  B3.9

This is the handover, and it is a deliberate change of stance.

For thirteen lessons you have been the builder. Everything you added was a capability, and every decision was about making the system work. From the next lesson on, every one of those components is read by somebody trying to make it do something you did not intend — and every control you added is read as something with a bypass.

The same map, annotated differently

what you builtwhat it becomes in Function B
ingress accepting traveller textB1.2 — prompt injection
the vendor MCP serverB1.3 — indirect injection, through a document you fetched
tool descriptions you trustedB1.9 — a rug-pull after approval
memory with origin recordedB1.4 — what happens when the origin is dropped
agent-to-agent envelopesB1.7 — one injection reaching four agents
the per-action token exchangeB2.3 — and the systems that log it without enforcing it
the human gateB3.9 — at four hundred approvals a day
the audit trailB1.14 — and the question it still cannot answer

Nothing in that column is a criticism of the build. It is what the build looks like to somebody who wants it to fail, and a builder who has never seen their own system described that way ships the same defect in the next one.

Blast radius is the number you carry forward

Before you go on, measure one thing: what the agent you just built can reach and damage in a single run, if every instruction it followed were chosen by an attacker. That number decides how much autonomy it can be given, and it is the input to almost every decision in Function B.

2 · Measure what you just built can reach

The skill below computes reach and damage for a single run and maps that to an autonomy level. It is the last thing you do as the builder and the first number Function B argues with.

The skill

3 · Mark your own nine, and then do one small thing

Two tasks to close Function A, and an assistant cannot do either for you — that is deliberate. Everything up to here can be driven by an agent that reads the lesson and runs the skill; these two ask what you now know.

First, mark the list you wrote in A0.1. Back then you wrote the nine words down the side of a page and guessed which file each one would live in. Find that page.

  1. Get the finished application onto your computer. On a Mac or Linux:
python3 scripts/checkpoint.py --at A2.5 --out work/finished

On Windows, the same line with python and backslashes:

python scripts\checkpoint.py --at A2.5 --out work\finished
  1. Open the work/finished folder and look at the file names.
  2. Go down your list of nine and tick the ones you got right.
  3. Count the ticks.

All nine words now have a file, and you wrote every one of those files. The ones you got wrong, and the question marks you left, are the things you learned between then and now — say them out loud, because that is the part that sticks.

Second, a task nobody walked you through. Pick one:

  • Make the agent refuse an action it currently allows, and say which of the four refusal points you used — policy, human, verifier, or resource server — and why that one.
  • Add a tenth word to A0.1's nine: something CyberTravels has that the list does not name. Say which file it lives in.
  • Take the blast radius number above and reduce it by one irreversible action, without turning the feature off.

Then answer these three in your own words, out loud or in writing, without re-reading the pages:

  1. What can your agent do? Name the tools and who they act for.
  2. What stops it doing something it should not? Name the controls in the order a request meets them.
  3. What would you look at first if it did something wrong yesterday at 3am?

If any of the three is hard to answer, that is the signal — and the lesson to go back to is named in the answer you could not give. Function B assumes all three.

▶Real time execution as skill

The skill — skills/architecture/blast-radius-review/SKILL.md

name: blast-radius-review
description: >-
  Compute what an agent can reach and damage in a single run, and decide the
  autonomy level its blast radius can support. Use when reviewing an agent
  design or deployment, deciding whether an action needs human approval, sizing
  a sandbox, or answering how bad it would be if an agent were fully
  compromised.
allowed-tools: Read, Grep, Glob

Blast radius as a design metric

Blast radius is not an adjective. It is the set of resources an agent can change before anyone can stop it, and it is computed from three inputs:

blast_radius = reachable_resources × action_irreversibility × time_to_human_stop

Treating it as a number is what lets it be a design constraint instead of a discussion.

When to use this

At design review, before raising an agent's autonomy, and after any change that adds a tool, a credential, or a scheduled trigger.

Procedure

1 — Enumerate reachable resources. For each tool the agent can call, list what it can touch with attacker-chosen arguments — not what it touches in the happy path. A Bash tool with unrestricted arguments reaches everything the process can reach; record it that way rather than as one row.

2 — Grade irreversibility. Per action:

GradeMeaningExample
0read-onlyquery, list
1reversible with effortwrite a file, open a PR
2reversible only with a backupdelete a row, force-push
3irreversible or externally visiblesend an email, pay, publish, rotate a key

Grade 3 actions are the whole reason approval gates exist. An agent whose worst action is grade 0 does not need one.

3 — Measure time-to-human-stop. How long between the agent deciding and a human being able to intervene? Interactive with a prompt is seconds. A scheduled run at 03:00 with notifications off is hours. This term dominates the product more often than people expect, and it is usually the cheapest to fix.

4 — Place it on the autonomy ladder.

LevelMeaningRequires
L1suggests; human executesnothing
L2acts within a bounded sandboxreversible actions only
L2.5acts, but grade-3 actions need approvala working approval path
L3acts unattendeddemonstrated containment + audit + stop authority

An agent at L3 whose grade-3 actions are unbounded is misclassified, not brave.

5 — Find the cheapest reduction. Usually one of: remove a credential from the environment, split one broad tool into two narrow ones, add a choke point in front of the irreversible action, or shorten time-to-stop with a notification. Recommend the one with the best radius reduction per unit of friction, and say what it costs.

Example

Input — the fixture committed at the top of scripts/blast_radius_review.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.

Output — the opening lines of a real run:

action              reversible  external  routing         per day
delete_row          False       False     human approval  6
issue_refund        False       True      human approval  2
read_report         True        False     policy only     400
rotate_credential   False       False     human approval  1
send_email          False       True      human approval  3
update_ticket       True        False     policy only     260
write_draft         True        False     policy only     120

The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.

Output contract

{
  "resources": [{"tool": "str", "reachable": ["str"], "unbounded": false}],
  "actions": [{"action": "str", "irreversibility": 0, "why": "str"}],
  "time_to_human_stop_seconds": 0,
  "blast_radius": {"score": 0, "inputs": {"resources": 0, "max_irreversibility": 0, "seconds": 0}},
  "autonomy": {"current": "L1|L2|L2.5|L3", "supported": "L1|L2|L2.5|L3", "mismatch": false},
  "reductions": [{"change": "str", "new_score": 0, "friction": "low|medium|high"}]
}

Show inputs. A blast-radius score without its terms cannot be challenged, and an unchallengeable metric stops being used.

Failure modes

  • Counting the happy path. Enumerate with attacker-chosen arguments.
  • Ignoring time-to-stop because it is not about permissions. It is the term that separates an incident from a near miss.
  • Raising autonomy because the agent has been reliable. Reliability is not containment; it is the absence of an adversary so far.

Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.

1In your AI agent — you are taught

The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.

Install it, once. From the folder you cloned. --all does every agent it finds on your machine:

python3 scripts/install_skills.py --all --lessons A2.5

Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.

Then pick a2-5-what-you-have-built:

agentinstall just for this agentstart it
Claude Codepython3 scripts/install_skills.py --tool claude --lessons A2.5type /a2-5-what-you-have-built
GitHub Copilot (VS Code)python3 scripts/install_skills.py --tool copilot --lessons A2.5type /a2-5-what-you-have-built in Chat
Cursorpython3 scripts/install_skills.py --tool cursor --lessons A2.5type / in Agent chat and search for it
Codexpython3 scripts/install_skills.py --tool agents --lessons A2.5type $a2-5-what-you-have-built, or run /skills

2As one Python command — you just run it

No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.

python3 scripts/lesson.py A2.5

3Read the code, and the difference — nothing runs

Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.

python3 scripts/checkpoint.py --at A2.5 --out work/cybertravels
python3 scripts/checkpoint.py --at A2.5 --diff

On Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.

Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.

Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.

If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.

✓What you just proved

The set of objects one run can reach, the subset it can change, the irreversible actions among those, and the autonomy level that radius supports — which will be lower than the one you gave it. Then two things the skill cannot produce: your nine guesses marked against the finished tree, and your own answers to what the agent can do, what stops it, and where you would look first.

✎Your turn

Remove the human gate and recompute. The radius grows by exactly the irreversible actions, which is the argument for the gate stated as a number rather than as a principle.

→Where this leaves you

What you can do now. The harness around the loop: spans that join reasoning to actions, an audit trail measured against the four questions an investigation asks, an evaluation suite with intervals and a test for its own dilution, and the blast radius of the agent you built, as a number.

What you still cannot do. Everything you have measured, you measured against what you intended. Nobody has yet read this system as somebody trying to make it do something else — and every component you added is, from that side, a way in.

B1.0 — the same architecture, read adversarially, and the zero-trust rules that survive contact with agents.

Tools used

MITRE ATLAS

Skill source: skills/architecture/blast-radius-review

← PreviousA2.4 · Evaluating the agent's behaviour — tool calls, outputs, and the model as judge