All lessons › Function F — AI Governance for Agentic Systems › Building the Governance Framework — Risk and Control
F1.1

From framework control to key control indicator

F1
▶
Lightboard recording

Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.

Video placeholderpublishes automatically from recordings/F1.1.mp4
Risk

An annual review certifies nothing about a system that changed the week after it.

Control

Continuous assurance; control effectiveness redefined for probabilistic systems.

The lab

Change a prompt and show the control evidence going stale in real time.

◉Use case relevance

You tested the control in March and signed the assertion. The prompt changed in April, the model in May, and the tool scope in June. The assertion is still on file and has not described anything real since the day it was written.

At CyberTravels. KCI refuses a measurement that is not callable. That single constructor check is the lesson: a control indicator an engineer has to go and look up is taken once, at audit time, by somebody who knows what answer is wanted. Every KCI here reads the tree.

◴What this lesson is — Day 0, Day 1, Day 2

Turning a framework control into a key control indicator that can be computed this week without asking anybody, since the only alternative is a measurement taken once, at audit time, by somebody who knows what answer is wanted.

Day 0 — why

A control tested six months ago is not passing — it is unevidenced, and most tooling cannot represent that third state.

Day 1 — how

Turn a framework control into something computable, with a denominator and a target written before the measurement.

Day 2 — measure

The unit itself: a key control indicator. Everything else in this function produces one, feeds one, or reports one.

Measure this

what to measuredenominator or target
the unit itself — a key control indicatora computation, a denominator and a target; everything else in this function produces, feeds or reports one

▦The framework, and how it works

   march      test the control, sign the assertion
   april      the prompt changes
   may        the model version changes
   june       the tool scope changes
   december   the assertion is still on file

   point-in-time assurance for a system that changes between tests
   describes a system that no longer exists

Classical control testing has a simple shape: a control is designed, an auditor tests it once or twice a year, and a passing test is recorded for the period.

That works when the thing being tested changes only through a process that generates evidence. For an agent, the four things that change its behaviour are:

  • the model version — changed by your provider, possibly without notice,
  • the prompt — edited in a console,
  • the tool manifest — a config change,
  • the approval settings — a toggle in an admin UI.

None of them is a code change. None generates a change record. All of them invalidate the conditions the control was tested under.

The honest consequence is that a control tested six months ago is not passing — it is unevidenced, which is a third state most GRC tooling cannot represent. Introducing that third state is the first half of this lesson.

The second half is what you use instead, and it is the unit the rest of Function E is written in: a key control indicator. A number computed from the estate, with a denominator, and a target set before the measurement is taken. Section 5 below builds them from the framework controls in A0.0; every other lesson in this function either produces an indicator, supplies its denominator, sets its target, decides who owns it, or presents it as evidence to somebody outside the organisation. Each of them says which, in a line under its own concept.

2 · Demo — the same evidence, two ways of reading it

3 · Where it breaks — the two numbers those readings produce

4 · The control — a freshness window per control, derived from drift

The window is not an audit-calendar choice. It comes from how fast the thing the control tests actually changes.

5 · From a framework control to something you can compute

A control in a framework is written to be general, which is exactly what makes it untestable. "Accuracy, robustness and cybersecurity" (EU AI Act Art. 15) is an obligation, not a measurement. The work is turning it into a number the estate can produce on demand.

The test for whether you have finished is mechanical: can this be computed from the estate today, without asking anyone?

framework controlthe question it asksa KCI that answers it
EU AI Act Art. 15is it robust and secure?share of tool calls carrying provenance
EU AI Act Art. 12can you reconstruct it?share of agent runs with a complete trace
EU AI Act Art. 14is there human oversight?share of irreversible actions behind an approval
NIST AI RMF MEASUREdo you know how it performs?detections with a measured false-positive rate
OWASP Agentic T2can a tool be misused?object handlers that compare an owner
OWASP Agentic T3can privilege be escalated?agents operating inside declared scope

Three rules make the difference between a KCI and a metric somebody reports.

It names a denominator. "Twelve tool calls carried provenance" is a number; "41% of tool calls carried provenance" is an indicator. Without the denominator you cannot tell improvement from reduced traffic.

It has a target, written before it is measured. An indicator with no target cannot be met or missed, so it can only be discussed.

It is computed from the estate, not read from a register. A register records what somebody believed when they filled it in, which is the state the freshness window above says you cannot trust.

6 · The three ways a KCI goes wrong

It measures the instrument. If renaming a function moves the number, the indicator measures naming. This is the most common failure and the hardest to see, because the number looks responsive.

It has no denominator, so it drops when traffic drops. An indicator that improves during an outage is measuring the outage.

Every indicator in the set fails. A set that always reports a gap cannot be shown to discriminate, and a reader has no way to tell the instrument works from the estate being uniformly broken. Keep at least one the estate passes.

F1.13 computes six of these against the CyberTravels repository, and E5.4 re-measures them after a fix to decide whether the remediation actually restored anything — which is the whole reason for defining them here.

7 · What replaces the annual test, as a skill

If the window is short, something has to re-run inside it, and that something is an attestation: collect each control's verdict, resolve every evidence pointer, compute drift against the image digest and the tool manifest, and sign the result. Two rules in the procedure are the whole difference between an attestation and a slide — a missing verdict is not a pass, and a capped verdict does not get raised because the other evidence looked good. This is the file in this repository:

▶Real time execution as skill

The skill — skills/attestation/attestation-signer-lifecycle/SKILL.md

name: attestation-signer-lifecycle
description: >-
  Bundle control evidence, compute verdicts, and produce a signed verifiable
  attestation for a deployment, re-attesting on drift. Use to emit or verify
  a control attestation, to sign evidence into an in-toto envelope, or to
  decide whether drift requires re-attestation.
allowed-tools: Bash, Read, Write

Attestation Signer Lifecycle

Controls: All — the artifact

Use the existing framework

Do not invent a format. Wrap the evidence in an in-toto style statement inside a signed envelope, with the subject naming the deployment and its digests, and a typed predicate carrying the control verdicts. Express the predicate body in an assessment-results vocabulary so the verdicts map to control catalogues and an auditor can consume them.

The roles are worth naming explicitly: this skill set is the attester producing evidence, a separate service is the verifier appraising it against reference values, and the deployment gate is the relying party applying policy. Keep them separate — an attester that also decides whether it passed is not an attestation.

When to use this

Last, once every collector has run — and again on drift. Signing is a separate skill and a separate role on purpose: an attester that also decides whether it passed is not an attestation. Also use it to verify somebody else's envelope.

Procedure

  1. Collect every sub-skill verdict. A missing verdict is not a pass. If a skill did not run, the control is UNKNOWN and the attestation says so.
  2. Apply the confidence ceilings. Sandbox-egress and injection-screening are capped at PARTIAL by their own skills; the signer must refuse to raise them.
  3. Resolve every evidence pointer. A URI that does not resolve is a broken attestation, not a cosmetic issue.
  4. Attach framework mappings, so one artefact answers several catalogues.
  5. Compute drift. Compare image digest, configuration hashes and tool description hashes against the previous attestation. Any change triggers re-attestation — the tool-description hash specifically defends against a server mutating a tool after approval.
  6. Sign, and store alongside the image or in a transparency log.

Example

Input — the fixture committed at the top of scripts/attestation_signer_lifecycle.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.

Output — the opening lines of a real run:

control    age (days)   point-in-time   continuous
----------------------------------------------------
AC-1                3            PASS         PASS
AC-2                9            PASS         PASS
SB-1               45            PASS        STALE
SB-2              210            PASS        STALE
EV-1                5            PASS         PASS
DR-1                0            FAIL         FAIL

The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.

Output contract

{
  "subject": [{"name": "deployment_id", "digest": {"image": "sha256:…", "repo": "str"}}],
  "predicate_type": "str",
  "predicate": {
    "deployment_id": "str", "evaluated_at": "str", "evaluator_version": "str",
    "controls": [
      {"id": "C1_default_deny_least_privilege",
       "verdict": "PASS|FAIL|PARTIAL|UNKNOWN",
       "confidence": "HIGH|PARTIAL",
       "evidence": [{"skill": "str", "uri": "str", "hash": "str"}],
       "findings": ["str"]}
    ],
    "framework_mappings": {"owasp_llm": ["str"], "owasp_agentic": ["str"],
                           "atlas": ["str"], "nist_ai_rmf": ["str"], "iso_42001": ["str"]},
    "drift": {"since": "str", "changed": ["str"]}
  },
  "signatures": [{"keyid": "str", "sig": "str"}]
}

Failure modes

  • Reading a missing attestation as a pass. The relying party must fail closed: a signed file can be deleted, and absence is not evidence.
  • Raising a capped verdict because the other evidence looked good.
  • Signing without resolving evidence pointers, which produces a tamper-evident document full of dead links.
  • Attesting once. Without drift-triggered re-attestation the artefact describes a deployment that may no longer exist.

Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.

1In your AI agent — you are taught

The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.

Install it, once. From the folder you cloned. --all does every agent it finds on your machine:

python3 scripts/install_skills.py --all --lessons F1.1

Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.

Then pick f1-1-from-framework-control-to-key-control:

agentinstall just for this agentstart it
Claude Codepython3 scripts/install_skills.py --tool claude --lessons F1.1type /f1-1-from-framework-control-to-key-control
GitHub Copilot (VS Code)python3 scripts/install_skills.py --tool copilot --lessons F1.1type /f1-1-from-framework-control-to-key-control in Chat
Cursorpython3 scripts/install_skills.py --tool cursor --lessons F1.1type / in Agent chat and search for it
Codexpython3 scripts/install_skills.py --tool agents --lessons F1.1type $f1-1-from-framework-control-to-key-control, or run /skills

2As one Python command — you just run it

No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.

python3 scripts/lesson.py F1.1

3Read the code, and the difference — nothing runs

Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.

python3 scripts/checkpoint.py --at F1.1 --out work/cybertravels
python3 scripts/checkpoint.py --at F1.1 --diff

On Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.

Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.

Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.

If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.

✓What you just proved

The skill loads and reports its shape, and its failure modes are the governance lesson stated as engineering: the relying party must fail closed on a missing attestation, because reading absence as a pass is exactly the annual-test habit arriving in a new format. Drift against the digest and the manifest is what re-triggers it, not the calendar.

✎Your turn

Pick your three most important AI controls and set a freshness window for each from the observed change rate of what it tests. Then recompute your posture. The number will drop, and it will be the first honest one you have had.

Tools used

promptfoo

Skill source: skills/attestation/attestation-signer-lifecycle

← PreviousF1.0 · Start here — what AI governance means