This starts chapter E2. Its course guide is E2 in the Function E course guide →
Not recorded yet. The lesson below is complete and runnable today — the video is added later without touching this page.
recordings/E2.1.mp4Everything is indexed hot because nobody priced it, so retention is cut across the board and the traces go first.
A tiering decision per source, driven by the queries the SOC actually runs, with the cost of each tier stated.
Tier six sources against the queries that need them and compare the bill with index-everything.
The lake gets designed twice: once on a whiteboard where everything is indexed, and once when the invoice arrives and retention is cut across the board. The second design is the one that runs, and it is made by somebody who does not know which source a forensic replay needs.
soc/lake.py tiers each source by the queries the SOC actually runs. model_io is 900 GB queried twice a quarter and lands in cold; audit_rows is 8 GB and lands in hot. The bill is $267 against $2,972 all-hot — and the alternative to tiering is not a bigger bill, it is a retention cut applied to every source at once, with the traces going first because they are the newest.Tiering each telemetry source by the queries the SOC runs against it rather than by how important it feels, with the cost of each tier stated — because the alternative to a tiering decision is a retention cut applied everywhere at once.
Everything is indexed hot because nobody priced it, so retention is cut across the board and the agent traces go first.
Tier each source by the fastest query that reads it, and price the result against indexing everything.
29% cheaper on the sample estate — and, more importantly, the prompts that are 23% of the volume survive at 1% of the hot price instead of being deleted.
Measure this
| what to measure | denominator or target |
|---|---|
| telemetry cost under tiering | 29% cheaper on the sample estate |
| prompts retained rather than deleted | 23% of volume, at 1% of the hot price |
the queries decide the tier. nothing else does.
triage "what did this agent do in the last hour" seconds -> HOT
scope "everything this identity touched, 90d" seconds -> HOT
hunt "unexplained tool use over a fortnight" minutes -> WARM
forensics "reproduce one run, any time in a year" hours -> COLD
(none) nothing reads it never -> DROP
agent prompts: the largest source, read by ONE query, which can wait
priced hot -> the line that gets cut when the bill arrives
and E5.1 has nothing left to replay from
tiered cold -> survives at ~1% of the hot costEvery rule in this chapter is written against something. This lesson is that something, and it is designed twice if you are not careful: once on a whiteboard where everything is indexed, and once when the bill arrives and retention is cut across the board by whoever is holding the invoice.
The second design is the one you run, and it is made with no information about what the SOC actually asks.
Derive the tier from the queries. Write down what the SOC runs and how fast each needs an answer — triage in seconds, a hunt in minutes, a forensic replay in hours — then tier each source by the fastest query that reads it. Nothing else about the source decides it: not its volume, not how interesting it feels, not who asked for it.
| tier | answers in | what belongs there |
|---|---|---|
| hot | seconds | anything triage or scoping reads |
| warm | minutes | anything a hunt reads |
| cold | hours | anything only forensics reads |
| drop | never | anything no query reads at all |
That last row is the one people skip. A source no query reads is not cheap storage — it is a liability with a bill attached.
The saving is the headline. What the tiering protects is the point: agent prompts are the single largest source in most estates and are read by exactly one query, which can wait hours. Priced hot they are the line that gets cut, and cutting them removes the only thing E5.1 can replay a run from.
Anchor → E1.0. Detect is bounded below by what the lake can answer and how fast. A source tiered cold cannot serve a seconds-deep triage query however good the rule above it is — and a source deleted for cost sets that interval to infinity for everything that reads it.
The skill sorts six CyberTravels log sources by one question: how fast does the quickest query that reads this need an answer?
Then it prices two options against each other — keeping everything in fast storage, against moving some to cheaper slow storage.
The last thing it prints is the source somebody would have cut to save money, and what that would have cost you during an incident.
The skill — skills/detection/telemetry-tiering-cost/SKILL.md
name: telemetry-tiering-cost
description: >-
Choose hot, warm or cold storage for each log source from the fastest SOC
query that reads it, and price the result. Use when designing a detection
data lake, when a SIEM bill forces a retention cut, or when agent traces are
about to be deleted for costing too much.
allowed-tools: Read, Grep, GlobA detection lake gets designed twice. Once on a whiteboard, where everything is indexed and searchable, and once when the bill arrives, where retention is cut across the board by whoever is holding the invoice. The second design is the one you run, and it is made with no information about what the SOC actually asks.
Deriving the tier from the queries makes it one decision instead of two, and it is the difference between keeping agent traces and losing them.
Before building the lake, before a retention cut, and whenever a source is proposed for deletion on cost alone.
1 — Write down the queries the SOC runs, with the speed each needs. Triage in seconds. A hunt can take minutes. Forensic replay can take hours. If you cannot name the query a source serves, that is the finding.
2 — Tier each source by the fastest query that reads it. Nothing else about the source matters — not its volume, not how interesting it feels, not who asked for it. Seconds means hot, minutes means warm, hours means cold, and a source no query reads should be dropped rather than tiered.
3 — Price both designs. Index-everything-hot against the tiered version. The absolute rates vary by platform; the ratios between tiers do not, and the decision turns on the ratios.
4 — Look at what the tiering saves, and at what it protects. The saving is the headline and the protection is the point: the biggest, cheapest-to-cut source is usually agent prompts, and it is the only thing a forensic replay can run against.
index everything hot $ 18,993 / month
tiered by query $ 13,398 / month
difference $ 5,595 / month (29%)The run continues past this. test_skills.py executes the script on every build, with no model configured — so what CI proves is that it runs and refuses legibly, not that it prints these lines. The output above is a recorded run against a real model, and nothing re-diffs it: a model's answer is not reproducible, and this repository does not claim otherwise anywhere it can be checked.
{
"queries": [{"name": "str", "reads": ["str"], "needed_within": "seconds|minutes|hours"}],
"sources": [{"name": "str", "gb_month": 0, "retention_days": 0,
"tier": "hot|warm|cold|drop", "driven_by": "str"}],
"cost": {"all_hot": 0.0, "tiered": 0.0, "saved": 0.0},
"orphans": ["str"]
}agent-telemetry-retention).Three ways to do this lesson. Pick one. They run the same procedure against the same code — what changes is who drives it, and how much you see.
The agent reads the lesson, explains the idea, fetches the code, runs the check and answers your questions as you go. This is the one to take if you are here to learn.
Install it, once. From the folder you cloned. --all does every agent it finds on your machine:
python3 scripts/install_skills.py --all --lessons E2.1Then restart your agent and open it in that folder. On Windows, if python3 says Python was not found, use python instead.
Then pick e2-1-the-detection-data-lake:
| agent | install just for this agent | start it |
|---|---|---|
| Claude Code | python3 scripts/install_skills.py --tool claude --lessons E2.1 | type /e2-1-the-detection-data-lake |
| GitHub Copilot (VS Code) | python3 scripts/install_skills.py --tool copilot --lessons E2.1 | type /e2-1-the-detection-data-lake in Chat |
| Cursor | python3 scripts/install_skills.py --tool cursor --lessons E2.1 | type / in Agent chat and search for it |
| Codex | python3 scripts/install_skills.py --tool agents --lessons E2.1 | type $e2-1-the-detection-data-lake, or run /skills |
No agent, no conversation. The same lesson executes and prints the same readback. Take this one if you have no assistant, if you are reviewing rather than learning, or if you want the result without the teaching around it.
python3 scripts/lesson.py E2.1Neither of the above, and no model needed. The first command writes CyberTravels exactly as it stood at the end of this lesson; the second prints the diff — the lines this lesson added or changed, and nothing from any later one.
python3 scripts/checkpoint.py --at E2.1 --out work/cybertravels
python3 scripts/checkpoint.py --at E2.1 --diffOn Windows, type python wherever these say python3 — and use \ instead of / in the paths. Everything else is identical. A0.0 step 2 explains why.
Ways 1 and 2 need a model; way 3 does not. A signed-in Claude Code CLI needs no API key, and any OpenAI-compatible endpoint works too. With neither, the audit exits 2 and says so rather than inventing an answer. A0.0 sets this up.
Ways 1 and 2 both end with the same readback: What I did, What changed, The number and Read this next. A step that was skipped or refused is reported as exactly that, never as a pass.
If you did the last lesson’s challenge, expect this to stop. Most challenges ask you to edit the code, and ways 1 and 2 will not overwrite your edits: you get stopped, not a failure, with two options — --out for a fresh folder that keeps your work, or --force to discard it and start this lesson clean.
Four sources go hot because triage and scoping read them in seconds, host EDR goes warm, and agent prompts go cold — read by one query that can wait hours. Tiering costs about 29% less than indexing everything hot, and the prompts that are 23% of the volume survive at 1% of the hot price rather than being deleted.
List the five queries your SOC actually ran last month, then tier your sources from them. Any source that appears in no query is the finding — you are paying to store something nobody asks.
Tools used
Skill source: skills/detection/telemetry-tiering-cost