All lessons › Function D — Agentic Evaluation and Red Teaming › One Red-Team Lifecycle, End to End
D1.1

Platform ingestion and supply-chain risks

D1
▶
Lightboard recording

Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.

Video placeholderpublishes automatically from recordings/D1.1.mp4

The lab

Score a deployment's packages, hosted models and datasets for supply-chain risk, weighted by the authority the agent that loads them runs with.

◉Use case relevance

The first surface an attacker reaches is not the model — it is the hub the model was downloaded from. A squatted package name and a poisoned dataset both land before a single prompt is sent, and the platform accepted them because the name looked right.

At CyberTravels. redteam/ingestion.py inventories the four ways text somebody else wrote reaches your agents, with the column teams do not have: who can write here. Three of the four arrive labelled untrusted because B2.6 marks them. The fourth is knowledge/retriever.py's CORPUS — anyone who can add a travel template, no review, no label — and unlabelled() returns exactly that one.

◴What this lesson is — Day 0, Day 1, Day 2

Inventorying the paths by which text somebody else wrote reaches the agent, with the column an integration list does not have — who can write here, and does it arrive labelled. Including the path people miss, which is memory: an ingestion path with a delay.

Day 0 — why

The first surface an attacker reaches is the hub the model came from, and a squatted name is accepted because it looked right.

Day 1 — how

Assess each component on whether it can change without telling you, rather than on how popular it is.

Day 2 — measure

Components with a pinned digest, as a share of what the pipeline pulls at train or deploy time.

Measure this

what to measuredenominator or target
components with a pinned digestshare of what the pipeline pulls at train or deploy time

▦The framework, and how it works

   an attacker's first reach is the SUPPLY, not the model

   pip install helpf-ul-utils   squatted name   -> accepted
   model: org/whisper-turbo     never verified  -> accepted
   dataset re-pulled each train  changes freely  -> accepted

   assess on:  can this change without telling me?
     pinned library     no   -> low
     hosted model       yes  -> high
     re-pulled dataset  yes  -> high, and no digest = the finding

The first surface an attacker reaches is not the model — it is the hub the model was downloaded from. Dependency-squatting a package name, uploading a malicious model to a public registry, or shipping a poisoned dataset are all supply-chain attacks that land before a single prompt is sent.

Red teaming this surface means auditing the ingress filters: what a platform accepts, what it verifies, and what it takes on trust because the name looked right. The research discipline is to assess a component on whether it can change without telling you — a pinned library cannot, a hosted model can, and a dataset re-pulled on every train can change under a name that never did.

2 · Demo — score a supply chain the way an attacker reads it

3 · Where it breaks — the trusted name that was never verified

4 · The control — assess on mutability, not on popularity

5 · The procedure, as a skill

The skill does two things.

It scores every component CyberTravels pulls in. The question it asks about each one is simple: can this change without anybody telling us?

Then it sorts them into two piles. One pile is pinned to an exact version. The other can be swapped by a registry at any moment. The second pile is the one to worry about.

▶Real time execution as skill

The skill — skills/research/agent-supply-chain-assessment/SKILL.md

name: agent-supply-chain-assessment
description: >-
  Score new packages and MCP connectors for typosquatting and ordinary
  supply-chain signals, then re-score them weighted by the authority the agent
  that loads them runs with. Use when an agent installs its own dependencies or
  connects to a server somebody added last week.
allowed-tools: Read, Grep, Glob

The same package is a different risk inside an agent

Supply-chain assessment for agents differs in one term: the artefact runs with the agent's authority. A connector that trips three ordinary signals is a review; the same connector loaded by an agent holding production credentials is a block. Weighting by authority is what turns the ordinary assessment into the right answer.

When to use this

When an agent can install packages, when an MCP server is added, and at any review of what a coding agent is allowed to pull.

Procedure

1 — Establish the known-good set. The packages this project actually uses. Typosquat detection is a comparison against something; without the set it is a spell-check.

2 — Score edit distance against known-good names. A distance of one or two from a popular package, with a recent first-publication date, is the classic shape. Report the package it imitates, not just the score.

3 — Apply the ordinary signals. Age, maintainer count, download history, whether it was published after the agent asked for it, install scripts.

4 — Re-score weighted by agent authority. What credentials are in the environment the artefact will execute in, and what the agent can reach. This is the step that moves a review to a block and it needs no new information.

5 — Report the two verdicts side by side. Unweighted and authority-weighted. The difference is the argument for gating what agents may install, and it is easier to make with both numbers present.

Example

Input — the fixture committed at the top of scripts/agent_supply_chain_assessment.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.

Output — the opening lines of a real run:

requests==2.31.0        allow
requsts==2.31.0         block
      · unsigned — no attestation to source
      · published 3d ago — no soak time
      · only 12 downloads
      · distance 1 from popular package 'requests'
colourama==0.4.6        block
      · unsigned — no attestation to source

The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.

Output contract

{
  "known_good": ["str"],
  "artefacts": [{"name": "str", "kind": "package|mcp", "distance": 0, "imitates": "str|null",
                 "signals": ["str"], "verdict": "allow|review|block"}],
  "authority": {"credentials_present": ["str"], "reachable": ["str"]},
  "weighted": [{"name": "str", "verdict": "allow|review|block", "moved": true}]
}

Failure modes

  • Typosquat detection with no known-good set. Everything is close to something.
  • Assessing the package and not the environment. The authority is the term that differs.
  • Treating an MCP connector as configuration. It is code that runs with the agent.

Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.

1In your AI agent — you are taught

The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.

Install it, once. From the folder you cloned. --all does every agent it finds on your machine:

python3 scripts/install_skills.py --all --lessons D1.1

Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.

Then pick d1-1-platform-ingestion-and-supply-chain:

agentinstall just for this agentstart it
Claude Codepython3 scripts/install_skills.py --tool claude --lessons D1.1type /d1-1-platform-ingestion-and-supply-chain
GitHub Copilot (VS Code)python3 scripts/install_skills.py --tool copilot --lessons D1.1type /d1-1-platform-ingestion-and-supply-chain in Chat
Cursorpython3 scripts/install_skills.py --tool cursor --lessons D1.1type / in Agent chat and search for it
Codexpython3 scripts/install_skills.py --tool agents --lessons D1.1type $d1-1-platform-ingestion-and-supply-chain, or run /skills

2As one Python command — you just run it

No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.

python3 scripts/lesson.py D1.1

3Read the code, and the difference — nothing runs

Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.

python3 scripts/checkpoint.py --at D1.1 --out work/cybertravels
python3 scripts/checkpoint.py --at D1.1 --diff

On Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.

Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.

Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.

If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.

✓What you just proved

The pinned libraries score low-risk, the hosted model and the re-pulled dataset score high because neither can be pinned, and the one component with no provenance at all is the finding.

✎Your turn

List every model and dataset your pipeline pulls at train or deploy time. The ones with no pinned digest are the ones an attacker can change without touching your code.

Tools used

SigstoreOSV

Skill source: skills/research/agent-supply-chain-assessment

← PreviousD1.0 · Start here — the evolution of non-deterministic threat simulation