Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.
recordings/C2.3.mp4Pattern matching floods the queue; the false-positive rate is what actually changed.
Stage 7: deterministic rules for what rules do well, model reasoning for what rules cannot express.
Score grep, taint rules and model review against the same corpus, then combine them behind a confidence gate.
Weight access changes what you can see and what you can change. It turns questions about behaviour into questions about mechanism — and it is also how a supply-chain attacker turns a helpful model into a specifically unhelpful one.
appsec/sast.py, run against your own tools/ and agents/. It returns exactly five findings — the four yes rows of cybertravels/LABELS.md plus the library row — and none of the three IDOR-only rows. Clear sast.WRAPPERS and sync_vendor disappears: that one line is the whole of the library column, and the reason a mature codebase scans cleaner than it is.Two passes, and why the split follows from the defects rather than from taste: deterministic rules for what a pattern can express, a model for the class where the defect is the absence of a call and there is nothing to match at any ruleset width. Including the case in between — a house wrapper that makes every rule naming the library blind.
Pattern matching floods the queue, and the classes that matter in agentic code cannot be written as patterns at all.
Deterministic rules for what rules do well, then a model pass for what they cannot express.
The clearest before-and-after in the commons: Semgrep holds precision 1.00 and finds about one real defect in seven; the reasoning pass finds about six in ten. Four times the recall, same code — and the extra false positives are what the next four stages exist to remove.
Measure this
| what to measure | denominator or target |
|---|---|
| SAST precision and recall, deterministic pass | precision 1.00, recall about 1 in 7 on the sample |
| recall of the reasoning pass on the same code | about 6 in 10 — four times the recall, and the extra false positives are the next four stages' work |
what weight access gives you see interpretability probes, refusal boundaries, features change fine-tuning, distillation, adapters the same two verbs an attacker has: a poisoned adapter is a small file that changes behaviour in exactly one situation nobody tests
Stage 7 — Vulnerability auditing. The stage people mean when they say "SAST", and the one where the two halves of this pipeline are easiest to confuse with each other.
The deterministic half is a real scanner with real rules. Semgrep, CodeQL, OpenGrep. Parse the code to a graph, ask a rule a question about it, and get the same answer every time. That repeatability is what lets you gate a merge on it — a probabilistic check cannot block a build, because the same commit can pass on one run and fail on the next.
Its limit is not accuracy. On the file below Semgrep's precision is 1.00 at every ruleset width; it does not report bugs that are not there. Its limit is that a rule only finds the pattern somebody wrote, so its recall is a configuration decision — and one that is invisible, because a narrow scan and a wide scan both exit 0.
The probabilistic half reads the code and reasons. No rule needs to exist first, which is exactly its value, and it is the only thing that reaches a defect that is the absence of a call. It also invents defects, confidently, with a similar-looking confidence number attached.
So the two are not competing generations where the newer one wins. They answer different questions and fail in opposite directions, and this lesson runs each as its own skill:
| deterministic — Semgrep | probabilistic — the model pass | |
|---|---|---|
| same answer twice | yes | no |
| can gate a merge | yes | no |
| finds what no rule expresses | no | yes |
| typical failure | missed it entirely | reported it and it was not there |
| output is a | finding | hypothesis |
That last row is the load-bearing one. Everything the model says enters the pipeline as a hypothesis, and stages 8–12 are what turn one into a finding.
Where you are in the pipeline. `` [Ingestion & Mapping] ──> [Threat Modelling] ──> [Discovery] └─ stages 1-4 └─ stages 5-6 └─ stages 7-10 ──> [Dynamic Validation] ──> [Reporting] └─ stages 11-14 └─ stage 15 ``Everything this chapter scans is one tree: cybertravels/ — the reference architecture from B1.1, as source. Ingress, orchestrator, the four agents, the tools, both MCP servers, knowledge, messaging. egress/ is absent, because CyberTravels has no gateway.
Eight defects, enumerated by hand in LABELS.md before any scanner saw it — a key written afterwards is a description of the scan. Four correct functions are in the key too, because a corpus where everything is broken cannot measure precision.
# tools/bookings_api.py
def get_booking(session, booking_id): # 20 CWE-639 no owner check
def get_my_booking(session, booking_id): # the same read, authorised
def cancel_booking(session, booking_id): # 34 CWE-639 and it writes
def search_bookings(session, reference): # 41 CWE-89 + CWE-639
# tools/payments_api.py
def issue_refund(session, booking_id, amount): # 8 CWE-639 on the money path
def download_invoice(session, path): # 23 CWE-22 + CWE-639Look at get_booking and get_my_booking. They are four lines apart and almost identical; one of them is a critical finding. The difference is a call that is present in the second and absent in the first, and holding that in mind is the whole of this lesson.
Not a forty-line taint engine written to fit in a lesson. Semgrep 1.176.0, against that file, three configurations — run.sh reproduces all three and the raw JSON is committed beside the skill.
The third is a custom taint rule, which is what a real one looks like:
rules:
- id: cybertravels-sql-concat
languages: [python]
severity: ERROR
message: >-
Caller-controlled input is concatenated into a SQL string. Use a
parameterised query.
mode: taint
pattern-sources:
- pattern: |
def $F(..., $X, ...):
...
pattern-sinks:
- pattern: $CUR.execute(...)mode: taint is the whole difference between generations of scanner: not a better pattern, a different question — does caller-controlled input reach this sink?
The skill scores each of the three real runs against the six-defect key and reports precision and recall separately, because merging them into one "accuracy" number hides the only one that moves. Then it partitions what every width missed into the two classes that matter: a defect a rule could match and nobody wrote (write the rule) and a defect no rule can express (the next skill).
For twenty years the answer to "can a scanner find this?" was decided by one question: is there a pattern? Everything else followed. Rules were written per defect class and per library, coverage was a config decision, and a class with no syntax to match — a missing check, a wrong comparison, an absent expiry — was simply outside the tool.
That boundary moved. Systems that reason about a class rather than match a pattern now find defects that no rule expresses, and IDOR is the honest place to look at it: the class is defined by absence, so it is the case where pattern matching scores exactly zero and any number above that is new capability.
Semgrep published a benchmark for it in 2026 — 275 hand-reviewed labels across four repositories, the same revisions for every system (write-up):
| system | recall | precision | F1 |
|---|---|---|---|
| Semgrep Multimodal | 59.9% | 57.5% | 57.1% |
| Claude Security with Mythos | 13.9% | 80.1% | 23.7% |
| Codex Security | 11.3% | — | 17.7% |
Read it as a recall table, because that is what it is. The two bold cells are the argument. The most precise system on that board found roughly one IDOR in seven — it is right when it speaks and it stays quiet about six of every seven real defects. The best recall is six in ten, four times as many, from a system reasoning about the class.
Three things follow, and the third is the one that changes how you build the pipeline.
Precision is the easy half. A detector that reports nothing is perfectly precise. Any access-control tool that leads with its precision number is leading with the metric that improves when it finds less.
Recall needs a key, which is why nobody publishes it. You cannot compute what fraction of the real defects were found without knowing the real defects. That is expensive, it is manual, and it is the only thing that turns "the scan was clean" into a statement with content. This is why cybertravels/LABELS.md exists and why it was written before anything ran.
Six in ten is a real change and it is not a solved problem. It is the difference between a class you could not scan for at all and one you can scan for imperfectly — and at 57.5% precision it means roughly two in five reports are wrong. That is a hypothesis stream, not a finding stream, which is exactly how C2.4 and C2.5 are going to treat it.
One of them does not. Line 22 is a hardcoded key — lexical, and p/secrets was enabled and did not fire only because the string matches no known provider's format. That is a rule somebody writes this afternoon, and reaching for a model to find it is buying a language model to do a regex's job.
Line 7 is different in kind. There is no syntax for "this function should have called require_owner and did not", and there is no width of ruleset that reaches it — the defect only exists relative to the authority the caller holds, which is in a different file. That is the boundary, and it is narrow. Cross it deliberately and you have a reason to spend the model pass; cross it because the deterministic scan felt disappointing and you have bought noise.
The same repository, the class that has no pattern. The skill builds the denominator first — every unit that takes an identifier and touches a record — because recall without a denominator is not a number. Then it runs two detectors over it: what a rule can express, and the ownership comparison it cannot see.
Watch the row where two of the five were already in a scanner's output, for a different defect in the same function.
search_bookings concatenates its reference into SQL and returns every owner's bookings. Semgrep finds the first at the widest width. Nothing finds the second.
So the ticket says SQL injection in search_bookings, somebody parameterises the query, the finding closes, and the scan is green. The authorisation defect is untouched and now has no finding attached to it at all. The count went to zero and the risk did not move.
That is the failure mode this section exists for, and it is not exotic — it is what happens by default whenever one function carries a defect a rule can see and a defect it cannot. The fix is not a better rule. It is running the second detector over a denominator, and reporting recall against a key.
The same adapter every model-facing skill in this commons uses: offline a labelled replay, and against a served open-weight model the identical code. It is asked one question with a checkable answer, over the smallest slice in which the defect is decidable — the function, its signature, and the authority its caller holds.
The two functions it reviews are not retyped into the skill. They are cut out of cybertravels/tools/bookings_api.py with ast: get_booking, which the Semgrep run above missed at all three widths, and get_my_booking eleven lines below it, which does the same read and compares an owner. The second one is there because a review pass that is never wrong has not been tested.
In the offline run the replay returns 0.82 on the function with the defect and 0.77 on the one without it. Five points of confidence separate a real finding from a correct all-clear. It is tempting to read a threshold into a gap that size, and every pipeline that does it ships one.
Two reasons not to. The number is uncalibrated — 0.82 does not mean the claim is right 82% of the time, it means nothing in particular. And it is unstable: the same slice, the same model, ten runs, and the confidence moves further than the distance between your accept and reject bands. Sort a human's queue with it if you like. Do not let it decide anything on its own.
What kills a fabricated claim instead costs nothing and requires no judgement: the quoted line is not in the file. That check generalises, and C2.4 is where it becomes a stage.
A result from running this against a real model, worth keeping. The skill is written so the verifier is tested directly, against a quote known not to be in the slice — rather than by waiting for the model to fabricate one. That is not fastidiousness. An earlier version asserted that the model would invent a defect in the already-authorised control function, and against a served 7B open-weight model it did not: it read the function correctly and declined. The assertion failed because the model behaved well, which is a bug in the assertion. A pipeline check has to hold whether or not the model misbehaves on the day you run it.
Semgrep is cheap enough to run over the whole repository. The model pass is not — at four million lines it costs more than the finding is worth, and a model asked to review everything reviews nothing carefully.
So neither half is the interesting part. The allocation is, and the policy is three rules:
Get rule 2 wrong in the cheap direction and the authorisation defect on line 7 is never reviewed by anything, because the deterministic scan was green and nobody was surprised by that.
The skill — skills/appsec/sast-semgrep-deterministic/SKILL.md
name: sast-semgrep-deterministic
description: >-
Score a real Semgrep run against a ground-truth key and report recall per
ruleset width, not just the finding count. Use when choosing or reviewing a
SAST configuration, when a scan comes back clean, or when asked how much of
the codebase a scanner is actually looking at.
allowed-tools: Read, Grep, Glob, BashDeterministic SAST parses code to a graph and asks a rule a question about it. Given the same file and the same rules it returns the same findings, every time — which is what makes it the half of the audit you can gate a merge on.
The mistake is reading its output as a measure of the file. It is a measure of the rules that were enabled. A scan with one pack and a scan with seven disagree by a factor of four on the same file and both exit 0, so the number that matters is never the finding count — it is recall against a key, and almost nobody computes it because it requires knowing the answer in advance.
Before trusting any clean scan, and whenever a SAST configuration is chosen, widened or inherited. Also the honest way to compare two scanners: run both against a file whose defects you have enumerated by hand, and compare recall rather than volume.
1 — Enumerate the defects by hand first. One line per defect, with its CWE and whether it is expressible as a pattern at all. Do this before running anything; a key written after the scan is a description of the scan.
2 — Run the scanner at every width you are considering. The default pack, the widest set your team would plausibly enable, and any custom rules you wrote. Keep the raw JSON — the finding's rule id is what tells you which rule fired, and that is what you tune.
3 — Score each width against the key. Precision and recall separately. Deterministic SAST usually has excellent precision and the recall is the number under discussion; reporting them merged as "accuracy" hides exactly that.
4 — Partition what was missed into two classes. A defect a rule could match but none did is a coverage gap — someone writes the rule. A defect that is the absence of a call, or depends on the caller's authority, is not expressible as a pattern and no amount of rule-writing reaches it. Only the second class justifies a model pass.
5 — Report the width, not just the result. Every finding count is meaningless without the config that produced it. A report that says "Semgrep: 1 finding" and not "p/python + p/secrets" has withheld the part that decides whether the scan meant anything.
Input — the fixture committed at the top of scripts/sast_semgrep_deterministic.py. Edit it and re-run: the buckets, counts and verdicts below are derived from it, not hard-coded.
Output — the opening lines of a real run:
semgrep 1.176.0 · booking.py · 6 defects in the key, 5 of them expressible as a pattern
config found tp fp prec recall
p/python + p/secrets 1 1 0 1.00 0.17
seven registry packs 4 4 0 1.00 0.67
custom taint rule 2 2 0 1.00 0.33
Nothing about the file changed between those rows. Three of the fourThe run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"key": {"defects": 0, "pattern_expressible": 0},
"runs": [{"config": "str", "findings": 0, "true_positives": 0,
"precision": 0.0, "recall": 0.0}],
"missed_by_all": [{"line": 0, "cwe": "str", "class": "coverage-gap|not-expressible"}],
"widest_recall": 0.0
}class: not-expressible is the only honest argument for adding a probabilistic reviewer. If every miss is a coverage gap, the fix is a rule and it is cheaper.
The skill — skills/appsec/idor-detection-recall/SKILL.md
name: idor-detection-recall
description: >-
Find missing ownership checks on caller-supplied identifiers, and score the
result on recall against a labelled key rather than on finding count. Use when
auditing endpoints or tools that take a record id, when evaluating a scanner
that claims to detect broken object-level authorisation, or when a clean
access-control scan needs a number behind it.
allowed-tools: Read, Grep, GlobAn IDOR — broken object-level authorisation — is a function that takes an identifier from the caller, loads the record it names, and returns or mutates it without comparing an owner. Two things follow, and both change how you measure.
There is no pattern. get_booking and get_my_booking differ by one call that is present in the second and absent in the first. A rule matches syntax that is there; nothing matches syntax that is not. This is why the class survives every ruleset width in sast-semgrep-deterministic.
So precision is the easy half and recall is the whole problem. A scanner that reports nothing has perfect precision. The number that decides whether an access-control audit meant anything is what fraction of the real defects it found, and that requires a key — which is why almost nobody publishes one.
Semgrep did, in a 2026 IDOR benchmark: 275 manually reviewed labels across four repositories, the same revisions for every system.
| system | recall | precision | F1 |
|---|---|---|---|
| Semgrep Multimodal | 59.9% | 57.5% | 57.1% |
| Claude Security with Mythos | 13.9% | 80.1% | 23.7% |
| Codex Security | 11.3% | — | 17.7% |
Read the two bold cells together, because they are the shape of the problem. The most precise system found roughly one IDOR in seven. The best recall on the board is six in ten — from a system that reasons about the class rather than matching a pattern, and that is a real change from what rules could do. It is also not a solved problem, and a report that quotes only precision is quoting the number that improves when you find less.
On any surface where a caller passes a record identifier: REST handlers, RPC methods, and — the one people forget — agent tools, where the identifier arrives in a model-proposed argument and the tool runs with the agent's scope rather than the requester's. Also whenever an access-control scan comes back clean, because clean and unmeasured look identical.
1 — Enumerate the object-handling units first, not the findings. Every function that takes an identifier and touches a record. This is the denominator; without it you cannot compute recall and every later number is decoration.
2 — For each, find the ownership comparison, or its absence. A call that compares something from the record against something from the session. Trace it through helpers — a check inside require_owner counts, and a check in a decorator counts.
3 — Include the correct ones in the key. A corpus where everything is broken cannot measure precision. The near-identical safe twin of each defect is what catches a detector that flags on shape rather than on the missing check.
4 — Score recall and precision separately, and report the denominator. "Three findings" is not a result. "Three of the five object-handling units, against a key of eight" is.
5 — Weight by the authority the caller holds, not by the CWE. The same missing check is a different severity in a tool the Workflow Agent invokes with payments.refund than in one a traveller invokes with their own session. CVSS does not know that; your architecture map does.
Input — the fixture committed at the top of scripts/idor_detection_recall.py, scanning the cybertravels/ sample repository against cybertravels/LABELS.md.
Output — the opening lines of a real run:
cybertravels/ · 7 object-handling units, 4 of them authorised
detector found tp fp prec recall
pattern rule (execute + id) 0 0 0 0.00 0.00
ownership-comparison analysis 3 3 0 1.00 1.00The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"units": [{"file": "str", "unit": "str", "takes_id": true,
"ownership_check": "present|absent|via-helper"}],
"detectors": [{"name": "str", "found": 0, "true_positives": 0,
"precision": 0.0, "recall": 0.0}],
"findings": [{"unit": "str", "cwe": "CWE-639", "caller_authority": "str",
"severity": "str"}],
"denominator": 0
}denominator is the field that makes recall meaningful, and the one a scanner that reports only findings cannot give you.
@requires_owner above the function is an ownership check and a naive body scan misses it, reporting a false positive on correct code.session.user_id to the id the caller supplied rather than to the one on the loaded record passes every structural test and authorises nothing.download_invoice(session, path) is the same defect wearing a different parameter name.list_my_bookings takes no id at all — it is scoped by the session, so there is nothing to tamper with. Including it in the denominator understates recall.The skill — skills/appsec/sast-model-pass/SKILL.md
name: sast-model-pass
description: >-
Run a language model over a slice that deterministic rules structurally cannot
reach, and record what it says as a hypothesis rather than a finding. Use
after a rule-based scan has been scored, when a defect is the absence of a
call rather than the presence of a pattern, or when deciding whether an
unverified model confidence score may gate a pipeline.
allowed-tools: Read, Grep, GlobA rule matches syntax that is present. Some defects are the absence of something: no authorisation check, no ownership comparison, no expiry. There is nothing to match, so no ruleset finds them at any width, and that is the only defensible reason to put a model in an audit pipeline.
The price is that a model also reports defects that are not there, in the same tone, with a similar confidence number. So the output of this pass is not a finding list. It is a hypothesis list, and something else promotes a hypothesis to a finding.
After the deterministic scan is scored, on the files where the key says a defect exists and the rules were silent — never as a first pass over the whole repository. A model asked to review everything reviews nothing carefully, and at four million lines the pass costs more than the finding is worth.
1 — Take only the slice where the defect is decidable. The function, its signature, and the authority its caller holds. The signature is the part people drop and it is the part that decides the answer.
2 — Ask a question with a checkable answer. "Review this for security" returns an essay. "Does this function verify that the caller owns the record it returns — yes or no, and quote the line that does it" returns something you can check without a second opinion.
3 — Verify the quote against the file. If the model quotes a line, the line must exist. If it names a symbol, the symbol must be in the file. This costs nothing and kills the largest class of model error before a human sees it.
4 — Record every survivor as hypothesis, never finding. Carry the model, the prompt slice and the confidence with it, so the promotion decision downstream can be argued with rather than inherited.
5 — Do not gate on the confidence number. It is not calibrated and it is not stable across runs. Run the same slice ten times before you let any threshold into a pipeline; the variance is usually wider than the gap between your accept and reject bands.
Input — two functions cut out of cybertravels/tools/bookings_api.py with ast, not retyped: get_booking, which Semgrep missed at all three widths in sast-semgrep-deterministic, and its authorised twin get_my_booking eleven lines below it. Each is prefixed with the authority its caller holds, which is the one line no file contains.
Output — the opening lines of a real run:
get_booking
model says : MISSING CWE-639 at confidence 0.82
quoting : def get_booking(session, booking_id):
-> HYPOTHESIS: quote verified, status hypothesis (not a finding)
get_my_booking
model says : PRESENT CWE-639 at confidence 0.77
quoting : require_owner(session, row["owner_id"] if row else None)
-> NO CLAIM : reviewed, nothing asserted, nothing to promoteOne hypothesis from two functions. The control returning NO CLAIM is the half of the result that matters: a review pass that flags both is not detecting the defect, it is detecting that a function reads a booking.
The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"backend": {"kind": "replay|open-weight", "model": "str"},
"hypotheses": [{"unit": "str", "cwe": "str", "confidence": 0.0,
"quote_verified": true, "status": "hypothesis"}],
"rejected": [{"unit": "str", "why": "quote-not-in-file|symbol-absent"}],
"promoted": 0
}promoted is 0 here on purpose. This skill does not promote anything — deduplication, contextual verification, reachability and dynamic validation do, and a pipeline that lets the audit stage promote its own output has no independent step in it at all.
Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.
The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.
Install it, once. From the folder you cloned. --all does every agent it finds on your machine:
python3 scripts/install_skills.py --all --lessons C2.3Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.
Then pick c2-3-sast-for-agentic-code:
| agent | install just for this agent | start it |
|---|---|---|
| Claude Code | python3 scripts/install_skills.py --tool claude --lessons C2.3 | type /c2-3-sast-for-agentic-code |
| GitHub Copilot (VS Code) | python3 scripts/install_skills.py --tool copilot --lessons C2.3 | type /c2-3-sast-for-agentic-code in Chat |
| Cursor | python3 scripts/install_skills.py --tool cursor --lessons C2.3 | type / in Agent chat and search for it |
| Codex | python3 scripts/install_skills.py --tool agents --lessons C2.3 | type $c2-3-sast-for-agentic-code, or run /skills |
No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.
python3 scripts/lesson.py C2.3Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.
python3 scripts/checkpoint.py --at C2.3 --out work/cybertravels
python3 scripts/checkpoint.py --at C2.3 --diffOn Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.
Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.
Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.
If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.
Semgrep’s precision is 1.00 at all three widths and its recall is not: 0.12 on the default Python pack, 0.38 across seven registry packs, 0.25 on the custom taint rule — same tree, same engine, every scan exits 0. Five of the eight defects survive every width, in three classes: one coverage gap somebody fixes by writing a rule; one where the defect is textbook and the rule is written against requests while CyberTravels calls its own HTTP wrapper; and three IDORs that no pattern reaches at any width. The IDOR skill then builds the denominator — seven units that take an identifier and touch a record, two of them correctly authorised — and scores two detectors on it: the pattern rule finds 0 of 5, the ownership-comparison analysis finds 5 of 5. Two of those five were already in a scanner’s output for a different defect in the same function. The model pass then finds the missing-authorisation defect at 0.82 confidence, recorded as a hypothesis, and its claim about the already-authorised control function is rejected for quoting a line that is not in the file.
Two things, and the second is the one people skip. Run Semgrep against one of your own repositories at your current ruleset and at seven packs, and count the difference — whatever that number is, it has been the number all year. Then point the model pass at a real GLM-4.6 or Kimi K2 through Ollama and run one slice ten times. The variance in what it reports, and in its confidence, is what decides whether you can gate on confidence at all, and you cannot learn it from one run.
Tools used
Skill source: skills/appsec/sast-semgrep-deterministic