The skill — skills/grc/model-risk-validation-scope/SKILL.md

name: model-risk-validation-scope
description: >-
  Check whether a model validated under one autonomy level and tool set is still
  covered by that validation as deployed, and find the monitoring that stops at
  the model's output. Use when applying model risk management to an agentic
  system.
allowed-tools: Read, Grep, Glob

The validation was of a system that had no tools

Model risk management carries three pillars — validation, monitoring, governance — and each makes an assumption that agentic deployment breaks. Validation assumes the thing validated is the thing deployed; a model validated with no tools at L1 and deployed with three tools at L3 is the same model and a different system, and the validation does not cover it.

When to use this

Applying an existing model risk framework to agents, and at every autonomy or tool-manifest change afterwards.

Procedure

1 — Write down what was validated. Model, version, tool set, autonomy level, data scope. Validation scope is usually recorded as the model alone, which is the defect.

2 — Write down what is deployed, in the same fields. Then diff. Any difference in tools or autonomy means the validation does not cover the deployment, and that sentence is the finding.

3 — Check what monitoring observes. Most monitors the model's output distribution. List what it does not observe — rows written to production, actions taken, resources reached — because that is where agentic risk lives.

4 — Name the re-validation triggers. Tool added, autonomy raised, model version changed, data scope widened. Without triggers, re-validation happens on the audit calendar, which is the assumption that failed in the first place.

5 — Report per pillar. Validation coverage, monitoring blind spots, governance triggers. Three findings with three owners rather than one finding about the framework.

Example

Input — the fixture committed at the top of scripts/model_risk_validation_scope.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.

Output — the opening lines of a real run:

pillar                  what it asks                                  what it quietly assumes
conceptual_soundness    is the method appropriate for the purpose     assumes the purpose is stable and stated
independent_validation  did someone other than the builder check      assumes the thing checked is the thing deployed
ongoing_monitoring      is it still performing as validated           assumes performance is what changes

All three survive contact with AI. The assumptions are what break.
validation still covers what is deployed: False
   tool surface changed: ['db_update', 'read_ticket', 'write_ticket']

The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.

Output contract

{
  "validated": {"model": "str", "version": "str", "tools": ["str"], "autonomy": "str", "data": ["str"]},
  "deployed": {"model": "str", "version": "str", "tools": ["str"], "autonomy": "str", "data": ["str"]},
  "covers": false,
  "differences": ["str"],
  "monitoring": {"observes": ["str"], "does_not_observe": ["str"]},
  "revalidation_triggers": ["str"]
}

Failure modes

• Recording validation scope as the model. It is the system.
• Monitoring output distribution only. The actions are the risk.
• Calendar re-validation. The triggers are events, not dates.

The skill — skills/research/weight-access-capability-delta/SKILL.md

name: weight-access-capability-delta
description: >-
  Enumerate the capabilities local weights grant that a hosted API does not, and
  convert a rate limit into an attempt budget to show what removing it does to a
  low-probability technique. Use when assessing open-weight release risk or
  arguing about hosted safeguards.
allowed-tools: Read, Grep, Glob

The rate limit was the control

Hosted access and local weights differ in two ways that matter and one that gets all the attention. The capability delta — fine-tuning away refusals, reading activations, sampling without a filter — is real. The larger one is arithmetic: a hosted rate limit caps attempts, and removing it turns a technique with a 0.5% success rate into tens of thousands of successes in a day.

When to use this

Assessing the risk of an open-weight release, comparing hosted and local deployment, or evaluating a safeguard that exists only in the serving layer.

Procedure

1 — List the capabilities local access grants. Weight modification, activation access, unfiltered sampling, arbitrary system prompts, no logging. For each, say whether a hosted API offers it.

2 — Turn the rate limit into an attempt budget. Requests per minute × the window. Do the same for local inference from its throughput. Report the ratio; it is usually two or three orders of magnitude and it is the headline.

3 — Apply a per-attempt success rate. A technique that works 0.5% of the time produces a handful of successes hosted and tens of thousands locally. The technique did not change.

4 — Assess readiness for each safeguard. Which safeguards survive weight access at all — the ones in the serving layer do not. Say so plainly rather than listing them as mitigations.

5 — Weight the blast radius by scope. A capability that reaches a team's resources is not the same as one that reaches a run's own. Score tools by scope and mark which are gated.

Example

Input — the fixture committed at the top of scripts/weight_access_capability_delta.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.

Output — the opening lines of a real run:

setting                              attempts in 24h
------------------------------------------------------
hosted API, 20 req/min                        28,800
hosted API, 20 req/min, 5 keys               144,000
local open weights, 1 GPU                    691,200
local open weights, 8 GPUs                 5,529,600

ratio: 192× more attempts, with no abuse signal reaching anyone.

The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.

Output contract

{
  "capabilities": [{"name": "str", "hosted": false, "local": true}],
  "attempts": {"hosted_per_day": 0, "local_per_day": 0, "ratio": 0},
  "technique": {"per_attempt_rate": 0.0, "hosted_successes": 0, "local_successes": 0},
  "safeguards": [{"name": "str", "layer": "serving|weights", "survives_local": false}],
  "blast": {"tools": [{"name": "str", "scope": "self|project|team", "gated": false}], "score": 0}
}

Failure modes

• Listing serving-layer safeguards as mitigations for a weight release.
• Comparing capability and ignoring throughput. Throughput is the bigger term.
• Assuming a low success rate is a control. It is, until the attempt budget changes.