Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.
recordings/A2.4.mp4"The agent is 90% accurate" with no denominator, no matcher named and no judge validation. Three different numbers can be quoted from one run, the most flattering is the easiest to compute, and an unvalidated judge is a second unmeasured model grading the first.
Exact-match tool-call accuracy reported with its n and beside the two weaker matchers, output accuracy with unscoreable runs in their own column, and a judge whose agreement with ground truth is published before its verdicts are.
Score six agent runs three ways, then check the judge against the runs whose answer you already know.
Two teams report their agent at ninety per cent. One counted whether the right tool was named; the other counted whether the right tool was called with the right arguments, in the right order. Same runs, same agent, and one of those numbers is hiding a refund of 1400 where 140 was owed.
What it covers. The three surfaces of one agentic run that can be scored separately — the tool-call trajectory, the answer, and the answers no recorded truth reaches — and which of the three actually needs a model. Exact-match tool-call accuracy against the two weaker matchers that score higher on the same runs, output accuracy with unscoreable runs kept out of the denominator, and a model used as judge with a rubric, an undetermined verdict and a validation pass against the labelled subset.
Why a security engineer needs it. Every later function quotes an agent number at somebody: a recall figure in Function C, an attack success rate in D, a detection rate in E, a control indicator in F. All of them are this lesson's problem in a different costume — a rate is not a measurement until the matcher, the denominator and, where a model graded it, the judge's agreement with ground truth are stated beside it. The control it builds is the habit of publishing those three before the number they qualify.
"The agent is 90% accurate" survives every meeting it is said in, because nobody in the room can ask accurate at what, over how many runs, matched how, and graded by whom.
Score six recorded runs on tool-call accuracy and output accuracy without a model, then use one as judge for the run neither reaches, and validate it against the runs whose answer you already know.
Exact-match tool-call accuracy with its n beside it, and the gap to the two weaker matchers. Output accuracy with unscoreable runs counted separately rather than as passes. And the judge's agreement with ground truth, published before any verdict it gave.
Measure this
| what to measure | denominator or target |
|---|---|
| tool-call accuracy, exact match on name, arguments and order | rate, with n stated beside it |
| gap from exact match to name-only and to order-ignored | delta per matcher; each forgives a different real failure |
| output accuracy | correct / (correct + incorrect); unscoreable runs counted in their own column |
| judge agreement with ground truth | rate over the labelled subset, with false passes and false fails |
ONE RUN, THREE THINGS YOU CAN SCORE
the run what scores it needs a model?
+------------------------+
| tools it called, | exact match on no
| with arguments, | --> name + args + order,
| in order | and n beside it
+------------------------+
| the answer it gave, | compare with the no
| where truth was | --> recorded truth.
| recorded | unscoreable is its
+------------------------+ own column, not a pass
| the answer it gave, |
| where nobody | --> a model, as judge YES
| recorded a truth | + a rubric that
+------------------------+ allows "undetermined"
|
v
validate the judge on the rows that
DO have a truth. agreement, false
passes, false fails. an unvalidated
judge is a second unmeasured model.A2.3 built a suite and showed a score rising because the suite got easier. This lesson is the other half: what, exactly, is being scored on one run.
An agentic run is not one thing that is right or wrong. It is a trajectory of tool calls and an answer, and those fail independently. An agent can take the approved path and answer wrongly. It can answer correctly having taken a path no reviewer would have signed off. And the whole thing can be graded by a model that agrees with itself rather than with the truth.
So there are three surfaces, and only the third needs a model:
The strict question is: did it call the right tools, with the right arguments, in the right order. Three matchers answer three different questions, and the gap between them is the whole point:
| matcher | what it accepts | what it hides |
|---|---|---|
| exact — name, arguments, order | nothing else | this is the one to quote |
| name only | right tool, wrong arguments | refund(1400) where 140 was owed |
| order ignored | right calls, wrong sequence | rebooking before releasing the seat, so the traveller holds two |
All three are correct arithmetic. Only one is an answer to "is the agent doing the job". A number with no matcher named beside it is not comparable to anything, including itself next quarter — and n is part of the metric, because 90% of six runs and 90% of six hundred are different claims.
Where a correct answer was recorded, compare against it. The trap is the runs where nobody recorded one. They are not passes and they are not failures; they are unscoreable, and they get their own column. Folding them into the denominator as passes is the most common way an accuracy figure is inflated, and it is invisible in the result — the number simply looks better.
Some answers have no recorded truth and are still judgeable by a competent person: was this invoice explanation right, was that refusal appropriate. That is what a judge is for, and it is a real technique — but it is a model grading a model, so three rules make it worth reading.
Give it a rubric, not a vibe. The criteria in words, with examples of each verdict.
Let it say undetermined. A judge offered only pass and fail will answer confidently on cases it cannot decide. The third option is what converts a guess into a datum you can act on.
Validate it against the labelled subset — and publish that first. Run the judge over the runs where you do know the answer, and report agreement, false passes and false fails. This is the step that is almost always skipped, and it is the one that decides whether any of the judge's other verdicts mean anything. An unvalidated judge is a second unmeasured model, and you now have two problems.
Two more things a judge does that you have to control for: it prefers longer, more confident answers, and it prefers output from its own model family. Name the judging model in the report, always, for the same reason every finding in this commons names the model that produced it.
Cost per run, latency, and whether the task should have been attempted at all. Those are real and they are not accuracy; F3.1 and F3.5 pick that up as indicators. Saying so in the output is part of the procedure here.
Six recorded runs of CyberTravels' Workflow Agent, scored three ways. The runs are committed at the top of the skill's script, so you can change one and watch which matcher stops noticing.
Two of the six are the interesting ones. R2 refunds 1400 where 140 was owed — the right tool, the wrong argument, which name-only matching scores as a pass. R4 books the new Berlin seat before releasing the old one, so the traveller holds two — the right calls in the wrong order, which order-ignored matching scores as a pass. Each weaker matcher forgives a different real failure, which is why reporting one of them alone is worse than reporting neither.
R5 is the run with no recorded truth: whether an invoice explanation is right for German VAT is a judgement, and it is the only one of the six that needs a model at all.
This skill prints its deterministic half before it asks for a model, which no other skill in the commons does. That is deliberate: two of the three metrics are arithmetic over recorded runs, and a reader who meets the no-model refusal with no numbers above it learns the wrong lesson — that measuring an agent needs an AI. It does not. Only judging the unscoreable run does.
python3 skills/research/agent-eval-scoring/scripts/agent_eval_scoring.pyWith no endpoint you get the tool-call table, the output-accuracy line, and then exit code 2 saying what to set. With one, the same numbers and then the judge.
The skill below is the whole method: score the trajectory, score the outputs, judge only the remainder, then check the judge against the rows whose answer you already knew. Its output contract is what makes the last step checkable rather than asserted — judge_validation is a required key, so a report that skips it breaks the contract.
The skill — skills/research/agent-eval-scoring/SKILL.md
name: agent-eval-scoring
description: >-
Score an agent's runs on tool-call accuracy and output accuracy, then use a
model as judge only for what those cannot reach — and validate the judge
against a labelled subset before believing it. Use when asked how good an
agent is, or when an eval reports a single number with no denominator.
allowed-tools: Read, Grep, Glob"The agent is 90% accurate" is not a measurement until you say accurate at what. An agentic run has at least three scoreable surfaces and they fail independently: it can call the right tools and answer wrongly, answer correctly having taken a path nobody would approve, or do both well and be scored by a judge that agrees with itself rather than with the truth.
Before quoting any agent accuracy figure, when choosing between two models or two prompts, and whenever an evaluation uses a model to grade a model.
1 — Score tool calls against the expected trajectory. Exact match on the tool name and its arguments, in order. Report n beside the rate.
2 — Report the two weaker matchers next to it. Name-only, ignoring arguments; and order-ignored, treating the trajectory as a set. Both score higher than exact match on the same runs. Quoting either one alone is how a tool-call number becomes meaningless — a refund for the wrong amount calls refund correctly.
3 — Score outputs where ground truth exists. Exact or normalised match against the recorded correct answer. Count what you could not score separately and never fold it into the denominator as a pass.
4 — Use a judge only for what remains. Free-text answers with no ground truth. Give the judge the question, the answer and an explicit rubric, and make undetermined a verdict it is allowed to return — a judge with two options will invent confidence.
5 — Validate the judge against the labelled subset. Run it over the runs you do have ground truth for and report agreement, false passes and false fails. A judge you have not validated is a second unmeasured model.
6 — State what none of the three measured. Cost, latency, and whether the task should have been attempted at all.
Input — the fixture committed at the top of scripts/agent_eval_scoring.py. Edit it and re-run: the rates, counts and verdicts below are derived from it, not hard-coded.
Output — the opening lines of a real run:
tool-call accuracy n=6
exact (name + args) 0.500
name only 0.667 <- +0.167 for free
order ignored 0.667 <- +0.167 for free
output accuracy correct 2 incorrect 3 unscoreable 1 = 0.400
Neither number above needed a model. Only the unscoreable run does.The two weaker matchers score the same and are not forgiving the same run: name-only passes the refund of 1400 instead of 140, and order-ignored passes the rebooking that held two seats. Either one quoted alone hides a different real failure.
The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"tool_call_accuracy": {"exact": 0.0, "name_only": 0.0, "order_ignored": 0.0, "n": 0},
"output_accuracy": {"correct": 0, "incorrect": 0, "unscoreable": 0, "accuracy": 0.0},
"judge": [{"run": "str", "verdict": "pass|fail|undetermined", "why": "str"}],
"judge_validation": {"labelled_n": 0, "agreement": 0.0, "false_pass": 0, "false_fail": 0},
"headline": "str",
"not_measured": ["str"]
}n is part of the metric.undetermined. Forced into a binary it will answer confidently on cases it cannot decide.Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.
The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.
Install it, once. From the folder you cloned. --all does every agent it finds on your machine:
python3 scripts/install_skills.py --all --lessons A2.4Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.
Then pick a2-4-evaluating-the-agents-behaviour:
| agent | install just for this agent | start it |
|---|---|---|
| Claude Code | python3 scripts/install_skills.py --tool claude --lessons A2.4 | type /a2-4-evaluating-the-agents-behaviour |
| GitHub Copilot (VS Code) | python3 scripts/install_skills.py --tool copilot --lessons A2.4 | type /a2-4-evaluating-the-agents-behaviour in Chat |
| Cursor | python3 scripts/install_skills.py --tool cursor --lessons A2.4 | type / in Agent chat and search for it |
| Codex | python3 scripts/install_skills.py --tool agents --lessons A2.4 | type $a2-4-evaluating-the-agents-behaviour, or run /skills |
No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.
python3 scripts/lesson.py A2.4Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.
python3 scripts/checkpoint.py --at A2.4 --out work/cybertravels
python3 scripts/checkpoint.py --at A2.4 --diffOn Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.
Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.
Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.
If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.
Tool-call accuracy 0.500 on exact match over six runs, against 0.667 for both weaker matchers — and the two extra passes are different runs, so neither weak matcher is merely a looser version of the other. Output accuracy 0.400, from two correct and three incorrect, with the sixth run in its own unscoreable column rather than in the denominator. R2 fails both surfaces, which is the point: the run the weak matchers forgive is the run that was wrong. Then the judge's verdicts, and before you read any of them, its agreement with the five runs whose answer was already recorded. A judge that disagrees with those is not telling you anything about the sixth.
Change R2's expected amount in the fixture from 140 to 1400 so the trajectory is now right, and re-run. Exact match rises to 0.667 and name-only does not move, because it was already scoring R2 as a pass. Order-ignored rises too — to 0.833 — and that is worth understanding before you read it as a contradiction: it compares arguments as well as order, so R2's wrong amount was failing it for a reason that had nothing to do with sequence. Then check output accuracy, which does not move at all: you changed the approved trajectory, not the recorded truth, so the agent still told the traveller the wrong number. Three matchers, three different answers to "did it get better".
Tools used
Skill source: skills/research/agent-eval-scoring