Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.
recordings/F1.5.mp4Accepting a vendor's best-of-k demo as assurance; mistaking schema conformance for accuracy.
Read an eval report properly: execution-verified results, reliability across all attempts, trajectory scoring, judge independence.
Take the C2.19 scoring output and turn it into an evidence pack — then find the three ways the same numbers could mislead you.
An eval result is the closest thing this field has to evidence, and most eval results are unusable as evidence: no provenance, no retention, and no way to reproduce the run they came from.
evidences() reads an eval result and separates what it supports from what it does not. A best-of-8 demonstration shows the system can produce the answer; a control needs the rate. And conformance reported where accuracy was asked for is refused outright — it runs at about 100%, it is structural, and C2.18 caps two controls at PARTIAL for the same reason.What an evaluation result is and is not evidence of: a best-of-k demonstration, a rate with no interval, a score against tuned-on cases, and conformance reported where accuracy was asked for.
A vendor's best-of-k demo is not assurance, and schema conformance is not accuracy.
Report accuracy against a held-out key, with the sample size and an expiry.
An eval result read as an indicator: sample size is the denominator, the pass bar is the target, the expiry stops it ageing into a claim.
Measure this
| what to measure | denominator or target |
|---|---|
| an eval result read as an indicator | sample size is the denominator, the pass bar is the target, the expiry stops it ageing into a claim |
an eval result becomes evidence when it carries +---------------------------------------------------+ | what was run (harness + version + commit) | | against what (corpus + digest) | | when, by whom, and reproducible how | | retained where, for how long | +---------------------------------------------------+ a score with none of these is a screenshot
Evaluation output is the strongest audit evidence an AI programme can produce, and it only works if you present the right number.
C2.19 established the distinction; this lesson turns it into evidence:
Four properties make an eval result auditable:
Miss the fourth and you have produced a number that will be quoted three years from now about a system that has since had six model upgrades.
Anchor → F1.1. An eval result is an indicator reading — the sample size is its denominator, the pass bar is its target, and the expiry is what stops it aging into a claim. Read it that way and the conformance/accuracy split stops being a presentation choice: conformance has no denominator worth quoting, so it is not an indicator at all.
| the claim, as written | what it measures | defensible? |
|---|---|---|
| Our AI security harness scores 100%. | conformance | no |
| Our harness achieves 100% schema conformance. | conformance | yes |
| Our harness scores 0.81 expert accuracy on a 24-question held-out set. | accuracy | yes |
| Our harness passes all automated checks. | unspecified | no |
An evaluation result becomes evidence only when it tested the control that is claimed, ran on the system that is deployed, and states its failure mode. Most evidence fails the second.
The contract carries a field most packs would rather not have: conformance_reported. Setting it true should be read as a defect in the evidence, not a feature of it.
The most common overstatement in automated assurance, and it is usually made in good faith.
The skill — skills/grc/control-evidence/SKILL.md
name: control-evidence
description: >-
Turn evaluation and monitoring output into audit evidence that survives
challenge, and say plainly what it does not prove. Use when asked to evidence
a control, prepare for an audit, map evals to a control framework, justify
continuous control verification, or interpret a passing test as assurance.
allowed-tools: Read, BashAn evaluation result is not evidence until three things are true: it tested the control that is claimed, it ran on the system that is deployed, and its failure mode is stated. Most "evidence" fails the second.
Point-in-time control testing fails for AI systems specifically because the thing under test changes between tests — the model, the prompt, the tool set, and the data all move independently of any release.
Preparing audit evidence, mapping evals to controls, or reviewing whether a green dashboard supports the assurance being claimed from it.
1 — State the control claim precisely. "The agent cannot exfiltrate customer data" is not testable. "Egress from the agent workload is denied to all destinations outside the allowlist, enforced at the gateway" is. Evidence is only as good as the sentence it supports.
2 — Bind evidence to the deployed artefact. Record the model version, the prompt/config hash, the tool allowlist, and the commit — for the system as deployed, not as tested in a branch. Evidence that cannot be bound to a deployed artefact proves something about a different system.
3 — State the population and the sample. How many cases exist, how many were tested, how they were chosen. A sample chosen by the team that built the control is not independent, and that has to be written down rather than implied.
4 — Separate operating effectiveness from outcome. "The gate ran on every request" (operating) is different from "no harmful action got through" (outcome). Auditors ask for the first; incidents are caused by the second. Give both, labelled.
5 — Report accuracy, never conformance. A pipeline whose output is schema-valid 100% of the time has demonstrated that its serialiser works. Conformance is near-free by construction; accuracy is the expensive, interesting number. Any evidence pack leading with a conformance percentage is overstating itself, and an auditor who notices will discount everything else in it.
6 — State the blind spots. What the eval cannot see: cases not in the corpus, failure modes not modelled, drift since the run. A control with stated limits is stronger evidence than one claiming none.
7 — Set the re-verification interval from the change rate. If the model can be updated weekly, annual testing evidences a system that no longer exists. Tie re-verification to the artefact's change events, not the calendar.
Input — the fixture committed at the top of scripts/control_evidence.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.
Output — the opening lines of a real run:
n 24
conformance 1.0000 ← structural. NOT a quality claim.
expert accuracy 0.8750 ← the number that evidences EV-2
EV-2 PASS
expert accuracy 0.8750 over 24 held-out questions; conformance 1.0000 reported separately; key never exposed to the harness
at +10d → PASS
at +45d → STALEThe run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"control": {"id": "str", "claim": "str", "testable": true},
"binding": {"model": "str", "config_hash": "str", "tools": ["str"],
"commit": "str", "matches_deployed": true},
"sample": {"population": 0, "tested": 0, "selection": "random|risk_based|convenience",
"independent": false},
"results": {"operating_effectiveness": 0.0, "outcome_effectiveness": 0.0,
"accuracy": 0.0, "conformance_reported": false},
"blind_spots": ["str"],
"reverification": {"trigger": "on_model_change|on_config_change|periodic",
"interval_days": 0},
"conclusion": {"supports_claim": true, "limits": "str"}
}conformance_reported: true should be treated as a defect in the evidence pack, not a feature of it.
Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.
The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.
Install it, once. From the folder you cloned. --all does every agent it finds on your machine:
python3 scripts/install_skills.py --all --lessons F1.5Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.
Then pick f1-5-evaluation-output-as-audit-evidence:
| agent | install just for this agent | start it |
|---|---|---|
| Claude Code | python3 scripts/install_skills.py --tool claude --lessons F1.5 | type /f1-5-evaluation-output-as-audit-evidence |
| GitHub Copilot (VS Code) | python3 scripts/install_skills.py --tool copilot --lessons F1.5 | type /f1-5-evaluation-output-as-audit-evidence in Chat |
| Cursor | python3 scripts/install_skills.py --tool cursor --lessons F1.5 | type / in Agent chat and search for it |
| Codex | python3 scripts/install_skills.py --tool agents --lessons F1.5 | type $f1-5-evaluation-output-as-audit-evidence, or run /skills |
No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.
python3 scripts/lesson.py F1.5Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.
python3 scripts/checkpoint.py --at F1.5 --out work/cybertravels
python3 scripts/checkpoint.py --at F1.5 --diffOn Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.
Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.
Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.
If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.
Conformance is 1.0000 while expert accuracy lands around 0.81 on 24 held-out questions. Two of four sample claims are defensible. The EV-2 control test passes against a stated 0.80 threshold, is valid for 30 days, and reads STALE at 45 days. All five auditability checks pass.
Find an eval number your organisation has quoted, internally or externally, and determine which of the two it was. Then check whether it has an expiry. Most do not, and are still being cited.
Tools used
Skill source: skills/grc/control-evidence