All lessons › Function B — Securing AI Architectures › Securing the Architecture — Runtime and the Gateway
B3.6

Human approval that survives volume

B3
▶
Lightboard recording

Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.

Video placeholderpublishes automatically from recordings/B3.6.mp4
Risk

An approval queue at volume approves everything, and the risk register still records it as a control.

Control

Approval reserved for irreversible actions only, with machine-generated content labelled as such.

The lab

Route actions by reversibility and measure how many reach a human under each policy.

◉Use case relevance

Approval works for the rare and irreversible and fails for everything else. The design question is not whether to have a human in the loop — it is how few decisions you can put in front of them, so that each one gets read.

At CyberTravels. The human gate you built in A1.7 is still there and, at volume, still 100% covered while nobody reads anything. You add approval_load() to policy.py: approvals per reviewer per hour against what reading one actually takes. Sixty an hour reports saturated — and the coverage metric does not move.

◴What this lesson is — Day 0, Day 1, Day 2

Measuring an approval gate rather than enabling one — approvals per reviewer per hour against what reading one takes — and the trap that makes it necessary: coverage stays at 100% while review collapses, and the risk register records a control that has stopped being one.

Day 0 — why

An approval queue at volume approves everything, and the risk register still records it as a control.

Day 1 — how

Reserve approval for irreversible actions only, and label machine-generated content as such.

Day 2 — measure

Approvals per reviewer per hour, and the share of them that are genuinely irreversible actions.

Measure this

what to measuredenominator or target
approvals per reviewer per hourrate
approvals that are genuinely irreversible actionsshare of approvals

▦The framework, and how it works

   what reaches the human            what does not
   +-------------------------+       +------------------------+
   | irreversible            |       | reversible             |
   | above a value threshold |       | inside a budget        |
   | outside normal pattern  |       | matching prior approval|
   +-------------------------+       +------------------------+
        a few per day                    everything else

   the control is the filter, not the click

Mitigates: T10 Overwhelming Human-in-the-Loop · T15 Human Manipulation.

B1.15 showed approval collapsing under volume while still reporting 100% coverage. The fix is not a better reviewer or a nicer queue. It is sending fewer things.

Route by reversibility, because that is what a human is actually useful for:

  • Reversible, bounded — no approval. Policy from B3.1 decides, and the action can be undone if it was wrong.
  • Reversible, expensive to undo — no approval, but recorded prominently and sampled after the fact.
  • Irreversible or externally visible — approval, every time. Sending mail, paying, publishing, deleting without a backup, rotating a credential.

The test for whether your gate will hold is arithmetic, not intent: how many requests per day reach a human? If the answer is more than a person can consider properly, the control is already a click, and the number tells you so before the incident does.

The T15 half is one line of implementation and easy to skip: mark machine-generated content as machine-generated wherever a human reads it. A recommendation that arrives with institutional formatting recruits authority it has not earned. Labelling it does not stop anyone acting on it — it restores the scepticism they would apply to a colleague.

What this control closes. Sends fewer things to humans, so the ones that arrive are read. The test is arithmetic: how many per day reach a person.

2 · The control

3 · Deciding what needs a human, as a skill

Routing by reversibility only works if somebody has computed what each CyberTravels agent can actually reach and damage in one run. That is a blast-radius review, and its output is an autonomy level rather than an opinion: enumerate the reachable actions with attacker-chosen arguments, not the happy path, and include time-to-stop, because how fast you can halt a run is part of how much it can cost. This is the file in this repository:

▶Real time execution as skill

The skill — skills/architecture/blast-radius-review/SKILL.md

name: blast-radius-review
description: >-
  Compute what an agent can reach and damage in a single run, and decide the
  autonomy level its blast radius can support. Use when reviewing an agent
  design or deployment, deciding whether an action needs human approval, sizing
  a sandbox, or answering how bad it would be if an agent were fully
  compromised.
allowed-tools: Read, Grep, Glob

Blast radius as a design metric

Blast radius is not an adjective. It is the set of resources an agent can change before anyone can stop it, and it is computed from three inputs:

blast_radius = reachable_resources × action_irreversibility × time_to_human_stop

Treating it as a number is what lets it be a design constraint instead of a discussion.

When to use this

At design review, before raising an agent's autonomy, and after any change that adds a tool, a credential, or a scheduled trigger.

Procedure

1 — Enumerate reachable resources. For each tool the agent can call, list what it can touch with attacker-chosen arguments — not what it touches in the happy path. A Bash tool with unrestricted arguments reaches everything the process can reach; record it that way rather than as one row.

2 — Grade irreversibility. Per action:

GradeMeaningExample
0read-onlyquery, list
1reversible with effortwrite a file, open a PR
2reversible only with a backupdelete a row, force-push
3irreversible or externally visiblesend an email, pay, publish, rotate a key

Grade 3 actions are the whole reason approval gates exist. An agent whose worst action is grade 0 does not need one.

3 — Measure time-to-human-stop. How long between the agent deciding and a human being able to intervene? Interactive with a prompt is seconds. A scheduled run at 03:00 with notifications off is hours. This term dominates the product more often than people expect, and it is usually the cheapest to fix.

4 — Place it on the autonomy ladder.

LevelMeaningRequires
L1suggests; human executesnothing
L2acts within a bounded sandboxreversible actions only
L2.5acts, but grade-3 actions need approvala working approval path
L3acts unattendeddemonstrated containment + audit + stop authority

An agent at L3 whose grade-3 actions are unbounded is misclassified, not brave.

5 — Find the cheapest reduction. Usually one of: remove a credential from the environment, split one broad tool into two narrow ones, add a choke point in front of the irreversible action, or shorten time-to-stop with a notification. Recommend the one with the best radius reduction per unit of friction, and say what it costs.

Example

Input — the fixture committed at the top of scripts/blast_radius_review.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.

Output — the opening lines of a real run:

action              reversible  external  routing         per day
delete_row          False       False     human approval  6
issue_refund        False       True      human approval  2
read_report         True        False     policy only     400
rotate_credential   False       False     human approval  1
send_email          False       True      human approval  3
update_ticket       True        False     policy only     260
write_draft         True        False     policy only     120

The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.

Output contract

{
  "resources": [{"tool": "str", "reachable": ["str"], "unbounded": false}],
  "actions": [{"action": "str", "irreversibility": 0, "why": "str"}],
  "time_to_human_stop_seconds": 0,
  "blast_radius": {"score": 0, "inputs": {"resources": 0, "max_irreversibility": 0, "seconds": 0}},
  "autonomy": {"current": "L1|L2|L2.5|L3", "supported": "L1|L2|L2.5|L3", "mismatch": false},
  "reductions": [{"change": "str", "new_score": 0, "friction": "low|medium|high"}]
}

Show inputs. A blast-radius score without its terms cannot be challenged, and an unchallengeable metric stops being used.

Failure modes

  • Counting the happy path. Enumerate with attacker-chosen arguments.
  • Ignoring time-to-stop because it is not about permissions. It is the term that separates an incident from a near miss.
  • Raising autonomy because the agent has been reliable. Reliability is not containment; it is the absence of an adversary so far.

Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.

1In your AI agent — you are taught

The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.

Install it, once. From the folder you cloned. --all does every agent it finds on your machine:

python3 scripts/install_skills.py --all --lessons B3.6

Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.

Then pick b3-6-human-approval-that-survives-volume:

agentinstall just for this agentstart it
Claude Codepython3 scripts/install_skills.py --tool claude --lessons B3.6type /b3-6-human-approval-that-survives-volume
GitHub Copilot (VS Code)python3 scripts/install_skills.py --tool copilot --lessons B3.6type /b3-6-human-approval-that-survives-volume in Chat
Cursorpython3 scripts/install_skills.py --tool cursor --lessons B3.6type / in Agent chat and search for it
Codexpython3 scripts/install_skills.py --tool agents --lessons B3.6type $b3-6-human-approval-that-survives-volume, or run /skills

2As one Python command — you just run it

No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.

python3 scripts/lesson.py B3.6

3Read the code, and the difference — nothing runs

Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.

python3 scripts/checkpoint.py --at B3.6 --out work/cybertravels
python3 scripts/checkpoint.py --at B3.6 --diff

On Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.

Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.

Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.

If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.

✓What you just proved

Routing by reversibility sends 12 actions a day to a human instead of 792, which is inside what one reviewer can consider properly — so the gate holds rather than degrading into a click — and machine-generated output is labelled where a person reads it. The skill also reports its shape. The failure mode to carry into your own estate is the last one: raising an agent's autonomy because it has been reliable. Reliability is a measurement of the happy path; blast radius is a measurement of the worst one, and only the second bounds what an approval gate is for.

✎Your turn

Count how many approvals your agents generate daily and compare it with 25. If you are above it, decide which actions are reversible enough to be handled by policy instead — that list is usually most of them.

Skill source: skills/architecture/blast-radius-review

← PreviousB3.5 · Validating what comes back