Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.
recordings/C2.18.mp4Control claims are asserted in a spreadsheet and never bound to a deployment. Nobody can say which repo, image, role, identity, gateway and guardrail the claim was about, so it cannot be re-checked when any of them change.
Eleven skills scoped to one deployment_id, emitting an in-toto/DSSE attestation whose predicate carries per-control verdicts, evidence URIs, framework mappings and drift — with sandbox-egress and injection-screening capped at PARTIAL because their claims are not provable.
Run the control-intent analyser over ten real agent and MCP repositories and read the attestation it produces for each.
Point this pipeline at an agentic system and four of its stages quietly stop meaning anything: there is no call graph, the sink is a tool schema, the source is retrieved text, and the dependency is somebody else's running process. A green scan over an agent repository is a true statement about the Python and says nothing about what the agent will do.
appsec/attest.py binds every control claim in Functions A and B to one deployment_id, with an evidence URI a PASS cannot be issued without. Two controls are capped at PARTIAL and will raise if you try to pass them — sandbox egress is enforced outside the process and injection screening is a classifier whose failure mode is silence. drift() catches the case that matters: verdicts unchanged while the image digest moved underneath them.Binding control claims to a deployment so they can be re-checked rather than re-asserted: per-control verdicts with evidence URIs, framework mappings gathered once, drift against the last attestation — and two controls capped at PARTIAL because this pipeline cannot prove them.
Control claims live in a spreadsheet and are never bound to a deployment, so nobody can say which build they describe.
Scope every skill to one deployment identifier and emit a signed attestation whose predicate names what was checked.
Deployments with a verifiable attestation, and claims in the spreadsheet with no attestation behind them.
Measure this
| what to measure | denominator or target |
|---|---|
| deployments with a verifiable attestation | share of deployments |
| claims with no attestation behind them | count |
claim attestation
"we enforce least privilege" subject: deployment_id @ digest
| predicate: per-control verdicts + evidence
unbound signed, re-issuable on any change
the ceiling that keeps it honest
+------------------------------------------------+
| INTENT_EVIDENCED strongest static verdict |
| PARTIAL (capped) sandbox egress, injection |
| PASS not in the vocabulary |
+------------------------------------------------+Everything the pipeline has done so far assumed one thing about the code it was reading: that its control flow is in the code. A call graph exists, entry points can be enumerated, and a path from a source to a sink either exists or does not. Every stage from C2.3 to C2.17 stands on that assumption.
An agentic system breaks it. The control flow is decided at runtime, by a model, from text that is not in your repository. Point the same pipeline at CyberTravels' Workflow Agent and four of its stages quietly stop meaning anything:
| the stage | what it assumes | what an agentic system does instead |
|---|---|---|
| reachability (C2.5) | a call graph | the model picks the tool; there is no static edge to trace |
| SAST (C2.3) | dangerous sinks are functions | the sink is refund(booking_id, amount) — a schema, matching no rule |
| the source | untrusted input arrives at an entry point | it arrives as retrieved text, in no repository |
| SCA (C2.7) | dependencies are declared in a manifest | an MCP server is a third party's running process whose tool descriptions land in your context window |
Read those four rows together and the shape of the problem is clear. It is not that the tools are weak. It is that the artefact they analyse — source code — no longer contains the behaviour. A green pipeline over an agentic repository is a true statement about the Python and says nothing at all about what the agent will do.
Three specific things no static analyser can express, and each needs a stage that does not exist yet:
1 — The tool surface is data, and it is somebody else's. An MCP server's tool descriptions are fetched at connect time. They are not in your repository, they were not reviewed with your code, and a server can change one after review. There is no scanner finding for "the description changed"; there is only a diff nobody is taking.
2 — Composition, not vulnerability. read_booking is safe. post_to_vendor is safe. An agent holding both can move a traveller's card details to a vendor endpoint, and no rule matches either function because neither is defective. The unit of risk is a capability, and capability is a property of the set of tools an identity holds — which is a deployment fact, not a source-code one.
3 — Intent is the strongest static claim available. You cannot show from source that a sandbox holds, that egress is enforced, or that an injection screen works — those are runtime properties and C2.0's line puts them firmly on the right-hand side. What you can show is that somebody intended the control: an imported sandbox, a validated audience claim, a provenance tag. That is a smaller claim, and unlike the larger one it is true.
So the SDLC needs three things it has not had until now: tool-surface review as a build artefact, diffed like an API contract; capability analysis over the set of tools an identity holds, rather than vulnerability analysis over functions; and an artefact that carries intent evidence, is bound to one deployment, and can be re-issued when anything it describes changes. That last one is an attestation, and it is the rest of this lesson.
---
Every control claim in this pipeline has the same weakness: it is a sentence in a document, and nothing binds it to a running system. "We enforce least privilege" is true of some deployment, at some time, and there is no way to re-check it when the image, the role or the tool surface changes.
An attestation fixes the binding. It is a signed statement about a specific deployment, carrying per-control verdicts and the evidence behind each one, that can be re-issued whenever anything it describes changes.
Two design decisions carry the whole idea.
A deployment_id is the join key. It resolves to a manifest of content-addressed artefacts — repo at a commit, image by digest, IAM role, workload identity, gateway route, guardrail ID, downstream services. Without it an IAM finding, an identity entry and a gateway policy are three unrelated facts about three things that may not be the same system.
Eleven skills, not one. One resolver, nine collectors split along evidence-source boundaries (code, IAM, network, sandbox, identity, gateway, ingestion, risk register, entitlements), and one signer. They split there because each needs different API clients — and they stay separate because a single mega-skill produces context bloat and verdicts nobody can read.
Then the part that makes it honest. Not every control is equally verifiable:
| Control | Confidence | Why |
|---|---|---|
| D1 default-deny / least privilege | HIGH | policy documents plus observed usage are readable |
| C3 identity chain / OBO | HIGH | delegation is impossible without an actor token, so its presence is proof |
| C4 gateway routing | HIGH if egress is enforced below the application | otherwise an agent opens a socket and bypasses it |
| C2 sandbox / no egress | PARTIAL, capped | absence of a covert channel is not provable; DNS and object-storage bypasses are documented |
| C5 injection screening | PARTIAL, capped | detector presence is verifiable; adaptive attacks drive published defences back above 95% success |
A tool that reports PASS on C2 or C5 is wrong, and the cap belongs in the artefact rather than in a footnote.
Static analysis cannot show that a control holds. It can show that somebody intended it — an imported sandbox, a validated audience claim, a provenance tag. That is a smaller claim and a true one, so the analyser never emits PASS.
These are the verdicts the analyser in labs/attestation/control_intent.py produced against the five most-deployed open-source MCP repositories and five most-used agent frameworks, cloned at HEAD. Not a simulation — the counts below are what the scan returned.
Three findings worth more than the table.
An in-toto statement, subject-bound to the deployment, predicate in assessment-results vocabulary. The signer is a separate skill and a separate role — an attester that also decides whether it passed is not an attestation.
The skill — skills/attestation/agent-code-surface-analyzer/SKILL.md
name: agent-code-surface-analyzer
description: >-
Statically enumerate an agent or MCP server's declared tools, dangerous
actions and declared-versus-actual capability surface from its repository
or image. Use to inventory what a deployment can do before trusting what
it says it does, to check MCP tool annotations against the code, or to
baseline tool descriptions for rug-pull detection.
allowed-tools: Read, Grep, Glob, BashControls: Static basis for controls 1, 2 and 5
Every other control is about constraining capability. This skill establishes what the capability actually is, from the code rather than from the manifest — because the manifest is a claim made by the thing being audited.
Before trusting what a deployment says it does — a new MCP server, an agent framework you are adopting, a vendor integration. Also on a schedule against anything already deployed: tool descriptions are fetched at connect time and a server can change one after review, so the baseline this produces is what makes a rug-pull visible.
subprocess, os.system, exec, eval → process executionopen(...,'w'), file writes, shutil, os.remove → filesystem writeDELETE, DROP, TRUNCATE → destructive downstreamreadOnlyHint, destructiveHint, idempotentHint, openWorldHint) are hints, not guarantees — the specification says so explicitly, and a server can mislabel a destructive tool as read-only. Treat every annotation as a claim to verify, never as truth. Apply the spec's pessimistic defaults: an unannotated tool is assumed destructiveHint: true and openWorldHint: true.
readOnlyHint whose body writes a file is the highest-value finding this skill produces.tool_description_hash per tool. This is the rug-pull baseline: a server can change a tool's description after the client approved it, and the hash is what detects that.Input — the fixture committed at the top of scripts/agent_code_surface_analyzer.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.
Output — the opening lines of a real run:
C1_default_deny_least_privilege INTENT_EVIDENCED runtime verdict needs observed usage and the
C2_sandbox_no_egress PARTIAL absence of a covert channel is not provable
C3_identity_chain_obo INTENT_EVIDENCED 4 signals; enforcement not shown
C4_gateway_guardrails INTENT_EVIDENCED runtime verdict needs reachability testing f
C5_injection_screening PARTIAL detector presence is verifiable; robustness
PASS is not in the vocabulary. The strongest static verdict is
INTENT_EVIDENCED, and two controls cannot exceed PARTIAL at all.The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"deployment_id": "str",
"tools": [
{"name": "str", "capability_class": ["process|filesystem|network|credential|destructive"],
"annotations": {"readOnlyHint": "bool|null", "destructiveHint": "bool|null",
"idempotentHint": "bool|null", "openWorldHint": "bool|null"},
"annotations_present": true,
"declared_vs_actual": "match|understated|overstated",
"evidence": [{"file": "str", "line": 0, "sink": "str"}],
"tool_description_hash": "str"}
],
"dangerous_actions": ["str"],
"unannotated_count": 0,
"verdict": "PASS|PARTIAL|FAIL"
}PARTIAL, not a clean pass.Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.
The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.
Install it, once. From the folder you cloned. --all does every agent it finds on your machine:
python3 scripts/install_skills.py --all --lessons C2.18Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.
Then pick c2-18-agentic-ai-in-the-pipeline:
| agent | install just for this agent | start it |
|---|---|---|
| Claude Code | python3 scripts/install_skills.py --tool claude --lessons C2.18 | type /c2-18-agentic-ai-in-the-pipeline |
| GitHub Copilot (VS Code) | python3 scripts/install_skills.py --tool copilot --lessons C2.18 | type /c2-18-agentic-ai-in-the-pipeline in Chat |
| Cursor | python3 scripts/install_skills.py --tool cursor --lessons C2.18 | type / in Agent chat and search for it |
| Codex | python3 scripts/install_skills.py --tool agents --lessons C2.18 | type $c2-18-agentic-ai-in-the-pipeline, or run /skills |
No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.
python3 scripts/lesson.py C2.18Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.
python3 scripts/checkpoint.py --at C2.18 --out work/cybertravels
python3 scripts/checkpoint.py --at C2.18 --diffOn Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.
Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.
Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.
If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.
Five controls resolve to INTENT_EVIDENCED, PARTIAL or NO_INTENT_FOUND and never to PASS. Across ten real repositories and fifty control evaluations the analyser returns 30 INTENT_EVIDENCED, 16 PARTIAL, 4 NO_INTENT_FOUND and zero PASS — with one widely-deployed MCP server shipping no tool annotations at all, so all of its tool sites inherit the specification's destructive, open-world default.
Run the analyser against one agent or MCP server you actually deploy. The interesting output is not the verdicts — it is the controls that come back NO_INTENT_FOUND, because those are the ones nobody has started.
Tools used
Skill source: skills/attestation/agent-code-surface-analyzer