The Rubric

Module 0.3 · The Landscape · Master Course

15 minutes · Prerequisite: Module 0.1 + 0.2

The foundational anchor

Where the model's context actually goes in a typical production session:

Tool outputs

67.6%

History + rest

18%

Tool defs

10.7%

System prompt

3.4%

Model logic

1.6%

Iterating on the system prompt (3.4%) while ignoring tool-output formatting (67.6%) is optimizing 3% and neglecting 68%. Modules 2 + 3 fix this.

The paradox and its resolution

A ~100-line Python agent scores 74–76% on SWE-bench Verified — only 4–6 points behind state-of-the-art systems from well-funded teams.

Apparent paradox: "the loop is all that matters"?

Resolution: the execution loop is NOT where the complexity lives. World-class teams invest 6–12 months in the 98.4% around the loop. The last 4–6 points, and the reliability/security/governance benchmarks don't measure, live entirely in the 98.4%.

"The loop doesn't matter" misunderstands worse than "the loop is all that matters." The loop is necessary and cheap. The 98.4% around it is where you earn your keep.

Decision audit, not feature checklist

Feature checklist (anti-pattern):
"Does it have a sandbox?"
"Does it have compaction?"
→ Output: a ranking. Tells you nothing a README couldn't.

Decision audit (the rubric):
"What pattern did they choose?"
"What did they GIVE UP?"
"Where in the code is the evidence?"
→ Output: understanding.

The tell: if your rubric output could come from a README, you didn't audit a decision.

"Has a sandbox" vs a real decision

Feature: "Pi has a loop, 4 tools, and a max-iterations guard." (Useless.)

Decision: "Pi chose ReAct + trust-the-model + no compaction, trading safety and long-session coherence for model-co-evolution and a 1,200-line codebase any engineer can read in an afternoon. Correct for a personal assistant; wrong for enterprise multi-tenant."

The second names a pattern, a tradeoff, and a use-case fit. That is what the rubric scores.

The twelve modules

#	Module	The decision it audits
1	Execution Loop	ReAct / plan-execute / graph / dumb-loop / conversation
2	Tool Design	static / dynamic / MCP; schema-first vs free-text; count
3	Context Mgmt	compaction / masking / JIT / notes / subagent delegation
4	Memory	in-context / files / semantic / episodic / DB
5	Sandboxing	inside vs outside; provider; fs/network scope
6	Permission	auto / deny-default / risk-tiered / plan-mode / flags / HITL
7	Error Handling	transient / LLM-recoverable / user-fixable / fatal
8	State & Checkpoint	git / super-steps / file / DB / none
9	Prompt Assembly	prompt size; tool injection; memory; truncation
10	Subagent Orch.	as-tools / handoffs / fork / teammate / worktree
11	Verification	computed / visual / model-judged / human
12	Observability	logs / traces / token acct / replay / metrics / diffing

The 6-phase methodology

Same pipeline for all 21 deep-dives.

1. First Contact      README · cloc · structure · system prompt (from source)
2. Architecture Map    loop diagram · tool list · trace one call · stop conditions
3. Decision Audit      12 modules · what & why · 3 agree · 3 disagree · cite file:line
4. Security Audit      credential flow · exec paths · injection · blast radius
5. Benchmark           published scores · token profile · cold start · compare
6. Score & Synthesize  /60 · Architect's Verdict · MLSecOps note · 3 best · 3 fix

Each phase produces an artifact that feeds the next. Phase 6 is the deliverable.

The deliverable format

Scoring sheet: 12 rows × (1–5 score + decision + tradeoff + code location). Total /60. Evidence required.

Architect's Verdict (exactly 3 sentences):
What does this harness optimize for? What does it sacrifice? Who should build on it?

MLSecOps Relevance (exactly 1 sentence):
What is the most important security property or vulnerability for offensive/defensive work?

The format constraint is the point — forcing every harness into the same template makes them comparable.

The 6 phases as a workflow

Per the visual stack: n8n first. The methodology is a process; n8n visualizes processes.

Trigger (pick harness)
   │
   ▼
Phase 1: First Contact ── README + cloc + structure
   │
Phase 2: Architecture Map ── grep loop/tools/stop-conditions
   │
Phase 3: Decision Audit ── grep evidence for 12 modules
   │
Phase 4: Security Audit ── grep exec/write/network/credential paths
   │
Phase 5: Benchmark ── README published scores + token profile
   │
Phase 6: Score & Synthesize ── emit /60 + Verdict + MLSecOps templates
   │
   ▼
Verdict artifact

The workflow does not score for you — a human or LLM judge fills the templates from the grep evidence. It operationalizes the method so no phase gets skipped.

Takeaways

Anchor: 67.6% tool outputs / 3.4% prompt / 98.4% harness / 1.6% model.
The rubric is a decision audit, not a feature checklist. Score what was chosen and what it cost.
6 phases, same for all 21 deep-dives. Phase 6 emits the deliverable.
The deliverable: /60 + 3-sentence Verdict + 1-sentence MLSecOps note.
Format constraint is the point — comparability turns deep-dives into a mental model.

Module 0 complete. Next: Module 1 — the execution loop, in depth.