Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.
recordings/F2.9.mp4Overclaiming control, or triggering a moratorium.
Explain bounded autonomy with evidence, and anticipate the real questions.
Defend one workflow in a mock supervisory conversation.
A supervisor can tell the difference between confidence and evidence. The hardest part of this conversation is framing genuine uncertainty without sounding like you have lost control of the estate.
prepare() reports accuracy and conformance as separate fields, puts known_gaps() at the top, and carries the three openings that actually get used. Naming the gaps first is not candour for its own sake — it is the only version of the conversation where the answers to those three are already written down.Preparing the assurance conversation as arithmetic rather than rhetoric: accuracy and conformance reported separately, coverage with the gaps named first, and the three openings that actually get used.
A supervisor who finds a weakness you did not disclose doubts everything else you said.
Bring the weakest number first, with its denominator and the gaps you already know about.
Coverage stated honestly, including stale and unevidenced controls. Volunteering the distinction is what makes the rest credible.
Measure this
| what to measure | denominator or target |
|---|---|
| coverage stated with stale and unevidenced controls included | share, with the distinction volunteered |
what a supervisor hears
"we are confident" -> on what evidence?
"we test continuously" -> show me a failure you caught
"we do not know yet, and -> credible
here is how we will know"
framing uncertainty without surrendering the roomConversations with regulators and auditors go well when you bring the number that is weakest and explain it, and badly when you bring the strongest and let them find the other one.
That is not a moral point, it is a practical one. A supervisor who discovers a weakness you did not disclose now doubts everything else you said, and the rest of the engagement is spent re-establishing credibility you had at the start.
Three things to bring, in this order:
The third one is the one people omit, and it is the one that most reliably converts scepticism into a working relationship.
Anchor → F1.1. A regulator conversation goes differently when bounded autonomy is presented as numbers that were computed rather than as a description of intent. Bring the indicator, its denominator, and the gaps you already know about.
| how you open | what it is | what happens next |
|---|---|---|
| “Our harness scores 100%.” | conformance quoted as quality | the follow-up “against what key?” ends the meeting's credibility |
| “We have full coverage of our AI controls.” | stale and untested counted as passing | one request for dates exposes it |
| “Conformance is 100%; expert accuracy is 50% against a held-out key of 20. Coverage is 63%, with SB-1 stale at 45 days and two controls not yet deployed.” | both numbers, honestly | nothing left for them to discover — the conversation moves to the plan |
Conformance 1.0000, expert accuracy 0.5000, against a held-out key. The skill reports both to the same precision, splits control coverage into evidenced, stale and unevidenced, and rehearses the three openings that get used.
The skill — skills/regulatory/assurance-conversation-prep/SKILL.md
name: assurance-conversation-prep
description: >-
Prepare for a regulator or auditor conversation by reporting conformance and
expert accuracy separately alongside control coverage, and rehearsing the
three openings that actually get used. Use before an assurance meeting.
allowed-tools: Read, Grep, GlobThe number that goes into an assurance conversation is usually conformance, because it is the one that is high. Conformance says the output had the right shape; accuracy says it was right. Reporting the first without the second is the failure mode that ends a meeting badly, and reporting both is what makes the rest of the conversation about controls instead of credibility.
Before any regulator, auditor or customer assurance conversation about an AI system.
1 — Compute conformance and accuracy separately, against a held-out key. Never the examples that were used for development. Report both to the same number of decimals so neither can be read as the other.
2 — Compute control coverage in three states. Evidenced, stale, unevidenced. A single percentage hides the stale bucket, which is the one you will be asked about.
3 — Rehearse the three openings. "Show me the evidence for this control." "What changed since the last test." "What would you not be able to detect." These are the ones that get used.
4 — Prepare the answer to the third one honestly. A stated blind spot is a programme that knows its own shape; a claim of full detection invites a search for the counterexample, and there is one.
5 — Bring the artefacts, dated. The single most common failure is a correct answer with the evidence three teams away.
Input — the fixture committed at the top of scripts/assurance_conversation_prep.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.
Output — the opening lines of a real run:
n 20
conformance 1.0000
expert accuracy 0.7500
control state
------------------------
AC-1 PASS
AC-2 PASS
SB-1 STALE The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"evaluation": {"conformance": 0.0, "accuracy": 0.0, "key": "held-out", "n": 0},
"coverage": {"evidenced": 0, "stale": 0, "unevidenced": 0, "of": 0},
"openings": [{"question": "str", "answer": "str", "artefact": "str|null"}],
"known_blind_spots": ["str"],
"artefacts": [{"control": "str", "artefact": "str", "as_of": "str"}]
}Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.
The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.
Install it, once. From the folder you cloned. --all does every agent it finds on your machine:
python3 scripts/install_skills.py --all --lessons F2.9Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.
Then pick f2-9-regulator-and-auditor-conversations:
| agent | install just for this agent | start it |
|---|---|---|
| Claude Code | python3 scripts/install_skills.py --tool claude --lessons F2.9 | type /f2-9-regulator-and-auditor-conversations |
| GitHub Copilot (VS Code) | python3 scripts/install_skills.py --tool copilot --lessons F2.9 | type /f2-9-regulator-and-auditor-conversations in Chat |
| Cursor | python3 scripts/install_skills.py --tool cursor --lessons F2.9 | type / in Agent chat and search for it |
| Codex | python3 scripts/install_skills.py --tool agents --lessons F2.9 | type $f2-9-regulator-and-auditor-conversations, or run /skills |
No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.
python3 scripts/lesson.py F2.9Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.
python3 scripts/checkpoint.py --at F2.9 --out work/cybertravels
python3 scripts/checkpoint.py --at F2.9 --diffOn Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.
Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.
Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.
If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.
Conformance is 1.0000 while expert accuracy is 0.5000 on a 20-question held-out key. Coverage is 5 of 8 with SB-1 stale and DR-1 and ST-1 unevidenced. The three openings show conformance-as-quality and inflated coverage failing on the first follow-up, and the generated disclosure script states both numbers, the stale and missing controls, and a date for each.
Generate this script from your own live control state rather than writing it. If it cannot be generated, your coverage number is being assembled by hand for each meeting — which is why it differs between meetings.
What you can do now. The same indicator set read as evidence: mapped to a horizontal regime, a sector overlay and a privacy position; a disclosure clock that runs on readings taken before the incident; and documentation that survives supervision because it is dated measurements rather than narrative.
What you still cannot do. You can now say what each indicator proves and to whom. You still cannot say which one to fix first, who is funded to fix it, or what happens when the board is shown a number nobody in the room can re-compute.
Chapter F3 runs the indicators as a programme, from the CISO office. Next → F3.1, translating agentic risk upward.
Skill source: skills/regulatory/assurance-conversation-prep