Module 0.3 — The Rubric

Course: Master Course — Harness Engineering (12+ Hours) Module: 0 — The Landscape Section: 0.3 Duration: 15 minutes Level: Senior Engineer and above Prerequisites: Module 0.1 (the three-job definition), Module 0.2 (the taxonomy + lineage)

Learning Objectives

After completing this section, you will be able to:

Distinguish a design decision audit from a feature checklist, and explain why only the former produces understanding.
State the foundational anchor (67.6% / 3.4% / 98.4% / 1.6%) and use it to justify where to invest engineering effort.
Apply the 12-module rubric to score any harness, with evidence, on a 1–5 scale per module.
Execute the 6-phase deep-dive methodology — the same method used for all 21 deep-dives later in the course.
Produce an Architect's Verdict and an MLSecOps Relevance note in the canonical template.

1. The Foundational Anchor

Before the rubric, one empirical anchor that justifies the entire framework.

In a typical production agent session, here is what the model actually sees in context:

Layer	Share of context
Tool outputs	~67.6%
Tool definitions	~10.7%
History + everything else	~18%
System prompt	~3.4%
Model decision logic (the "AI")	~1.6%

Hold those two bold numbers. If you have been iterating on your system prompt while ignoring how your tool outputs are formatted, you have been optimizing three percent of context while neglecting sixty-eight percent. That is the single most common senior-engineer mistake in this entire domain, and Modules 2 and 3 exist to fix it.

And one more number. A roughly 100-line Python agent scores 74–76% on SWE-bench Verified — only four to six points behind state-of-the-art systems built by well-funded teams. The MBZUAI study (May 2026) found ~98.4% of a production harness is infrastructure and ~1.6% is AI decision logic. Four independently-built agents converged on the same harness patterns.

Here is the resolution of the apparent contradiction, stated precisely, because it is the load-bearing claim of this module: the execution loop is not where the complexity lives. World-class teams invest six to twelve months building the infrastructure around that loop — context management, tool design, memory, verification, observability, security. The last four to six SWE-bench points, and more importantly the reliability, security, and governance properties that benchmarks do not measure, live entirely in the 98.4%.

A learner who concludes "the loop is all that matters" because the 100-line agent is competitive has misunderstood. A learner who concludes "the loop doesn't matter" has misunderstood worse. The loop is necessary and cheap. The 98.4% around it is where you earn your keep.

2. The Rubric Is a Decision Audit, Not a Feature Checklist

Most rubrics in software are feature checklists: "does it have a sandbox? does it have compaction? does it have multi-session?" Feature checklists produce a ranking. They do not produce understanding.

This rubric is different. It is a decision audit. Each of the twelve modules asks three questions of a harness:

What pattern did they choose? (ReAct loop? plan-then-execute? graph-based?)
What did they give up? (Every pattern trades something. ReAct is simple but compounds errors. Plan-then-execute is fast but brittle if the plan is wrong.)
Where in the code is the evidence? (A decision without a code location is an assertion, not an audit.)

The tell that you are doing it wrong: if your rubric output for a harness could be generated by reading its README's feature list, you have not audited a decision. You have transcribed marketing.

Consider the difference applied to sandboxing. "Has a sandbox" is a feature. "Puts the sandbox outside the agent's reach because credentials cannot live in-sandbox for multi-tenant workloads, accepting API-hop latency on every tool call in exchange for credential isolation" — that is a design decision with a stated tradeoff. The rubric scores decisions, not features. This is why Module 0.2's NemoClaw can inherit 90% of OpenClaw's code and still be a distinct, scorable harness: its entire score is in the governance decision, not the inherited features.

3. The Twelve Modules

The rubric has twelve modules. Each one is a "where does the complexity live" question. The first six go deep on a layer; the next six handle cross-cutting concerns. (Note: the numbering here follows the comparison-rubric source document, which maps closely but not identically to the course's Module 1–12 numbering. The course modules teach these decisions; the rubric scores them.)

#	Rubric module	The decision it audits
1	Execution Loop	ReAct vs plan-then-execute vs graph vs dumb-loop vs conversation-driven
2	Tool Design & Registration	static vs dynamic vs MCP; schema-first vs free-text; tool count
3	Context Management	compaction, observation masking, JIT retrieval, note-taking, subagent delegation
4	Memory Architecture	in-context vs working files vs semantic store vs episodic vs DB
5	Sandboxing & Isolation	inside-sandbox vs outside-sandbox; provider choice; fs/network scoping
6	Permission & Approval	auto-approve vs deny-default vs risk-tiered vs plan-mode vs per-flag vs HITL
7	Error Handling & Recovery	the four-category taxonomy (transient/LLM-recoverable/user-fixable/fatal)
8	State & Checkpointing	git vs LangGraph super-steps vs state file vs DB vs none
9	Prompt Assembly	system-prompt size; tool injection location; memory placement; truncation strategy
10	Subagent Orchestration	agents-as-tools vs handoffs vs fork vs teammate vs worktree vs sequential vs group chat
11	Verification & Feedback	computed vs visual vs model-judged vs human; tight vs staged vs loose loop
12	Observability & Debuggability	logs vs traces vs token accounting vs replay vs metrics vs session diffing

Note on numbering. The rubric's 12 modules do not map 1:1 to the course's 12 teaching modules. The rubric audits 12 decisions; the course teaches them across 11 modules (some rubric decisions are combined) plus a capstone. The mapping: rubric 1→Module 1, 2→Module 2, 3→Module 3, 4→Module 4, 5→Module 5, 6→Module 6, 7→Module 7, 8→Module 8, 9 (Prompt Assembly) is covered across Modules 1–3, 10 (Subagents) is covered in Module 1.3, 11 (Verification) →Module 9, 12 (Observability) →Module 10. Module 11 (Security Engineering) synthesizes the defense-in-depth stack across all rubric decisions. Module 12 is the capstone where you build and score against this rubric. When scoring deep-dives, score all 12 rubric modules — a 0/5 for "no subagents" is informative, not a gap.

Each module, in the full rubric reference (the "Harness Comparison — Rubric & Methodology" page in the knowledge base), has the same five-part structure: the decision, the known patterns, the tradeoffs, what to find in the code, and deep-dive questions. You do not memorize the patterns — you learn to recognize them and to name the tradeoff.

4. The 6-Phase Deep-Dive Methodology

The rubric tells you what to score. The methodology tells you how to score it, reproducibly, on any harness. Six phases.

Phase 1 — First Contact. Read the README top to bottom; note what is not explained. Run tokei or cloc; record lines by language. Map the file structure: entry point → loop → tools → context → memory → sandbox → permissions → error handling. Find and read the system prompt in full source, not docs. Note the license and any usage restrictions.

Phase 2 — Architecture Map. Draw the execution loop as a Mermaid diagram. List every tool shipped by default with its schema. Trace a single tool call from model output through dispatch, execution, result, and context insertion. Identify the context assembly function and draw its input stack. Identify all stop conditions.

Phase 3 — Design Decision Audit. For each of the twelve modules, answer: what pattern did they choose, and why? For each pattern, identify what they gave up. Find three decisions you agree with; explain why. Find three you would make differently; explain the alternative.

Phase 4 — Security Audit. Trace credential flow end to end. Identify all shell/exec/write/network paths. Test: can a tool output inject a new instruction that changes agent behavior? Check whether the harness logs tool calls with full inputs and outputs. Identify the blast radius of a compromised agent process. Search GitHub issues and CVEs for disclosures.

Phase 5 — Benchmark & Performance. Record all published benchmark scores and methodology (self-reported vs reproduced). Note the token usage profile: how expensive is a typical session? Run a comparable task on this harness and at least one other; document differences. Record cold-start time and time-to-first-tool-call latency.

Phase 6 — Score & Synthesize. Fill in the scoring sheet (1–5 per module, evidence required). Write the Architect's Verdict. Write the MLSecOps Relevance note. Document the three things this harness does better than any other studied. Document the three things you would fix if you forked it.

5. The Output: Scoring Sheet + Verdict

The deliverable of a deep-dive is a specific artifact, in a specific format. Memorize the format; you will produce it fourteen times.

The scoring sheet

Module	Score (1–5)	Key design decision	Tradeoff accepted	Code location
Execution Loop
(...10 more rows...)
TOTAL	/60

Scores are 1–5 with evidence required. A score without a code location is an opinion.

The Architect's Verdict (3 sentences)

What does this harness optimize for? What does it sacrifice? Who should build on it?

The MLSecOps Relevance (1 sentence)

What is the most important security property or vulnerability of this harness for offensive/defensive work?

This format is non-negotiable across all 21 deep-dives. The constraint is the point: forcing every harness into the same template makes them comparable, and comparability is what turns a pile of deep-dives into a mental model of the field.

6. Anti-Pattern: The Feature Checklist

The one anti-pattern to internalize before the lab. A feature-checklist rubric asks "does it have X?" A decision-audit rubric asks "what did choosing X cost?" The difference:

Feature checklist on Pi: "Pi has a loop, 4 tools, and a max-iterations guard. Score: minimal." (Useless. Tells you nothing a README couldn't.)
Decision audit on Pi: "Pi chose a ReAct loop with trust-the-model permission and no compaction, trading safety guarantees and long-session coherence for model-co-evolution and a 1,200-line codebase that any engineer can read in an afternoon. Correct for a personal coding assistant; wrong for enterprise multi-tenant. Score: 3/5 on loop (correct pattern, minimal observability), 2/5 on permission (deliberately thin), 5/5 on the fit between thickness and use case."

7. How to Score — The Scoring Guide

Each rubric module is scored 1–5. The score is not "how good is this feature" — it is "how well-reasoned is this decision, and is the tradeoff correct for the harness's stated purpose?"

The 1–5 scale:

Score	Meaning	Example (Loop module)
1	Absent or broken. No decision made.	No stop condition at all. The loop runs forever or until the context window fills.
2	Present but minimal. Decision made but no depth.	ReAct loop with max-iterations only. No error-recovery stop, no budget stop.
3	Solid. Correct decision for the use case. Adequate depth.	ReAct loop with all 5 stop conditions. Appropriate for a coding assistant.
4	Deep. Multiple stop conditions, error recovery, subagent delegation.	Claude Code's loop: 40+ permission flags, plan-mode, subagents, observability wrapper.
5	Best-in-class. The reference implementation others should study.	Tau's (DD-21) stateless loop with typed event union, steering queues, transcript repair.

Scoring requires evidence. A score without a code location (file:line or a specific architectural observation) is an opinion, not an audit. Every cell in the scoring sheet must have either a file reference or a specific, verifiable claim about the harness's behavior.

The fit judgment. A score of 2 is not always bad. Pi scores 2/5 on permission — deliberately. Pi's thesis is "trust the model, keep the harness thin." Scoring Pi 2/5 on permission and then saying "wrong for enterprise" in the verdict is the correct application of the rubric. The score measures depth; the verdict measures fit.

Worked scoring example — the three-stop-condition minimum:

A loop with fewer than three stop conditions scores at most 2/5. The minimum production set:

end_turn — the model says it's done.
max_turns — a hard cap (prevents infinite loops).
error_threshold — stop after N consecutive tool failures.

Missing any of these is a defect, not a design choice. A loop missing max_turns will run until the token budget is exhausted. A loop missing error_threshold will retry a failing tool indefinitely. These are not tradeoffs — they are minimums.

The second is what the rubric demands. The "score: 5/5 on fit" is not a rubric module — it is the meta-judgment the Architect's Verdict captures. The 12-module scores give you the components; the verdict synthesizes them into a defensible position.

7. The Course Map

This module closes Module 0. You now have the definition (0.1), the ecosystem map (0.2), and the rubric + methodology (0.3). Modules 1–11 each go deep on one of the twelve rubric decisions — loop, tools, context, memory, sandbox, permission, error handling, state, prompt assembly, subagents, verification, observability, security. Each module makes you fluent in one row of the scoring sheet. Module 12 is the capstone, where you build a harness and score it against your own rubric. And the 21 deep-dives apply this exact 6-phase method to the 🔴 Essential tier from Module 0.2.

8. Key Terms

Term	Definition
Decision audit	A rubric that scores what was chosen and what it cost, not whether a feature exists
Feature checklist	The anti-pattern rubric: "does it have X?" — produces rankings, not understanding
Foundational anchor	67.6% tool outputs / 3.4% system prompt / 98.4% harness / 1.6% model — justifies the rubric
6-phase methodology	First Contact → Architecture Map → Design Decision Audit → Security Audit → Benchmark → Score
Architect's Verdict	3-sentence synthesis: optimizes for / sacrifices / who should build on it
MLSecOps Relevance	1-sentence security note: most important property or vulnerability for offensive/defensive work
Scoring sheet	12 rows × (1–5 score + decision + tradeoff + code location), total /60

9. Lab Exercise: Score Pi End-to-End

Objective: Run all 6 phases against Pi and produce a complete scoring sheet + Architect's Verdict + MLSecOps Relevance note. This is the template for all 21 deep-dives.

Setup: GitHub Codespace or local. Clone Pi (or Aider as fallback per Module 0.1's lab note).

Steps:

Phase 1 (First Contact): README, cloc, file-structure map, system prompt from source.
Phase 2 (Architecture Map): Mermaid loop diagram; tool list with schemas; trace one tool call; identify stop conditions.
Phase 3 (Decision Audit): Fill the 12-row decision/tradeoff table. Cite file:line for each.
Phase 4 (Security Audit): Trace credential flow; list shell/exec/write/network paths; state the blast radius of a compromised Pi process.
Phase 5 (Benchmark): Note any published Pi benchmark; record a typical-session token estimate.
Phase 6 (Score): Complete the scoring sheet (/60), the 3-sentence Architect's Verdict, and the 1-sentence MLSecOps Relevance note.

Expected output: one pi-deep-dive.md containing all six phase outputs in the canonical template. This becomes your personal reference for the "thin harness" archetype — every later deep-dive compares against it.

Stretch goal: re-score Pi's rubric as if it were being deployed for multi-tenant enterprise (the Claude Code use case). Which scores change? Which tradeoffs flip from "correct" to "wrong"? That exercise — re-scoring the same harness for a different use case — is the real test of whether you understand thickness as a design decision.

10. References

Harness Comparison — Rubric & Methodology — the source document; the full 12-module decision audit and 6-phase methodology.
MBZUAI harness study (May 2026) — the 98.4% / 1.6% finding; convergence across four independently-built agents.
"Lost in the Middle" (Liu et al., 2023) — arXiv:2307.03172 — the 30%+ context-position performance drop; the empirical basis for the context-management rubric module.
Luzzardi, "Inside vs Outside Sandbox" (2025) — the framing behind the sandboxing rubric module.
SWE-bench Verified — the benchmark on which the 100-line agent scores 74–76%; the basis for "the loop is not where the complexity lives."
Yao et al., ReAct (2022) — arXiv:2210.03629 — the foundational pattern behind the Execution Loop rubric module.
Pi source — the lab harness for the end-to-end 6-phase exercise.
Tau (DD-21) — the educational reference harness; the clearest illustration of each rubric module in readable code (~300-line loop, typed events, non-destructive session tree). Use as companion when scoring deep-dives.
OWASP Agentic AI Top 10 (2026) — the risk taxonomy that Module 11 maps the defense-in-depth stack against. Each rubric module's security dimension connects to an ASI risk.

# Module 0.3 — The Rubric **Course**: Master Course — Harness Engineering (12+ Hours) **Module**: 0 — The Landscape **Section**: 0.3 **Duration**: 15 minutes **Level**: Senior Engineer and above **Prerequisites**: Module 0.1 (the three-job definition), Module 0.2 (the taxonomy + lineage) --- ## Learning Objectives After completing this section, you will be able to: 1. Distinguish a **design decision audit** from a feature checklist, and explain why only the former produces understanding. 2. State the **foundational anchor** (67.6% / 3.4% / 98.4% / 1.6%) and use it to justify where to invest engineering effort. 3. Apply the **12-module rubric** to score any harness, with evidence, on a 1–5 scale per module. 4. Execute the **6-phase deep-dive methodology** — the same method used for all 21 deep-dives later in the course. 5. Produce an **Architect's Verdict** and an **MLSecOps Relevance** note in the canonical template. --- ## 1. The Foundational Anchor Before the rubric, one empirical anchor that justifies the entire framework. In a typical production agent session, here is what the model actually sees in context: | Layer | Share of context | | --- | --- | | Tool outputs | **~67.6%** | | Tool definitions | ~10.7% | | History + everything else | ~18% | | System prompt | **~3.4%** | | Model decision logic (the "AI") | ~1.6% | Hold those two bold numbers. If you have been iterating on your system prompt while ignoring how your tool outputs are formatted, you have been optimizing three percent of context while neglecting sixty-eight percent. That is the single most common senior-engineer mistake in this entire domain, and Modules 2 and 3 exist to fix it. And one more number. A roughly 100-line Python agent scores 74–76% on SWE-bench Verified — only four to six points behind state-of-the-art systems built by well-funded teams. The MBZUAI study (May 2026) found ~98.4% of a production harness is infrastructure and ~1.6% is AI decision logic. Four independently-built agents converged on the same harness patterns. Here is the resolution of the apparent contradiction, stated precisely, because it is the load-bearing claim of this module: **the execution loop is not where the complexity lives.** World-class teams invest six to twelve months building the infrastructure *around* that loop — context management, tool design, memory, verification, observability, security. The last four to six SWE-bench points, and more importantly the reliability, security, and governance properties that benchmarks do not measure, live entirely in the 98.4%. A learner who concludes "the loop is all that matters" because the 100-line agent is competitive has misunderstood. A learner who concludes "the loop doesn't matter" has misunderstood worse. The loop is necessary and cheap. The 98.4% around it is where you earn your keep. --- ## 2. The Rubric Is a Decision Audit, Not a Feature Checklist Most rubrics in software are feature checklists: "does it have a sandbox? does it have compaction? does it have multi-session?" Feature checklists produce a ranking. They do not produce understanding. This rubric is different. It is a **decision audit**. Each of the twelve modules asks three questions of a harness: 1. **What pattern did they choose?** (ReAct loop? plan-then-execute? graph-based?) 2. **What did they give up?** (Every pattern trades something. ReAct is simple but compounds errors. Plan-then-execute is fast but brittle if the plan is wrong.) 3. **Where in the code is the evidence?** (A decision without a code location is an assertion, not an audit.) The tell that you are doing it wrong: if your rubric output for a harness could be generated by reading its README's feature list, you have not audited a decision. You have transcribed marketing. Consider the difference applied to sandboxing. "Has a sandbox" is a feature. "Puts the sandbox outside the agent's reach because credentials cannot live in-sandbox for multi-tenant workloads, accepting API-hop latency on every tool call in exchange for credential isolation" — that is a design decision with a stated tradeoff. The rubric scores decisions, not features. This is why Module 0.2's NemoClaw can inherit 90% of OpenClaw's code and still be a distinct, scorable harness: its entire score is in the governance decision, not the inherited features. --- ## 3. The Twelve Modules The rubric has twelve modules. Each one is a "where does the complexity live" question. The first six go deep on a layer; the next six handle cross-cutting concerns. (Note: the numbering here follows the comparison-rubric source document, which maps closely but not identically to the course's Module 1–12 numbering. The course modules teach these decisions; the rubric scores them.) | # | Rubric module | The decision it audits | | --- | --- | --- | | 1 | Execution Loop | ReAct vs plan-then-execute vs graph vs dumb-loop vs conversation-driven | | 2 | Tool Design & Registration | static vs dynamic vs MCP; schema-first vs free-text; tool count | | 3 | Context Management | compaction, observation masking, JIT retrieval, note-taking, subagent delegation | | 4 | Memory Architecture | in-context vs working files vs semantic store vs episodic vs DB | | 5 | Sandboxing & Isolation | inside-sandbox vs outside-sandbox; provider choice; fs/network scoping | | 6 | Permission & Approval | auto-approve vs deny-default vs risk-tiered vs plan-mode vs per-flag vs HITL | | 7 | Error Handling & Recovery | the four-category taxonomy (transient/LLM-recoverable/user-fixable/fatal) | | 8 | State & Checkpointing | git vs LangGraph super-steps vs state file vs DB vs none | | 9 | Prompt Assembly | system-prompt size; tool injection location; memory placement; truncation strategy | | 10 | Subagent Orchestration | agents-as-tools vs handoffs vs fork vs teammate vs worktree vs sequential vs group chat | | 11 | Verification & Feedback | computed vs visual vs model-judged vs human; tight vs staged vs loose loop | | 12 | Observability & Debuggability | logs vs traces vs token accounting vs replay vs metrics vs session diffing | > **Note on numbering.** The rubric's 12 modules do not map 1:1 to the course's 12 teaching modules. The rubric audits 12 *decisions*; the course teaches them across 11 modules (some rubric decisions are combined) plus a capstone. The mapping: rubric 1→Module 1, 2→Module 2, 3→Module 3, 4→Module 4, 5→Module 5, 6→Module 6, 7→Module 7, 8→Module 8, 9 (Prompt Assembly) is covered across Modules 1–3, 10 (Subagents) is covered in Module 1.3, 11 (Verification) →Module 9, 12 (Observability) →Module 10. Module 11 (Security Engineering) synthesizes the defense-in-depth stack across all rubric decisions. Module 12 is the capstone where you build and score against this rubric. When scoring deep-dives, score all 12 rubric modules — a 0/5 for "no subagents" is informative, not a gap. Each module, in the full rubric reference (the "Harness Comparison — Rubric & Methodology" page in the knowledge base), has the same five-part structure: the decision, the known patterns, the tradeoffs, what to find in the code, and deep-dive questions. You do not memorize the patterns — you learn to *recognize* them and to *name the tradeoff*. --- ## 4. The 6-Phase Deep-Dive Methodology The rubric tells you *what* to score. The methodology tells you *how* to score it, reproducibly, on any harness. Six phases. **Phase 1 — First Contact.** Read the README top to bottom; note what is *not* explained. Run `tokei` or `cloc`; record lines by language. Map the file structure: entry point → loop → tools → context → memory → sandbox → permissions → error handling. Find and read the system prompt in full source, not docs. Note the license and any usage restrictions. **Phase 2 — Architecture Map.** Draw the execution loop as a Mermaid diagram. List every tool shipped by default with its schema. Trace a single tool call from model output through dispatch, execution, result, and context insertion. Identify the context assembly function and draw its input stack. Identify all stop conditions. **Phase 3 — Design Decision Audit.** For each of the twelve modules, answer: what pattern did they choose, and why? For each pattern, identify what they gave up. Find three decisions you agree with; explain why. Find three you would make differently; explain the alternative. **Phase 4 — Security Audit.** Trace credential flow end to end. Identify all shell/exec/write/network paths. Test: can a tool output inject a new instruction that changes agent behavior? Check whether the harness logs tool calls with full inputs and outputs. Identify the blast radius of a compromised agent process. Search GitHub issues and CVEs for disclosures. **Phase 5 — Benchmark & Performance.** Record all published benchmark scores and methodology (self-reported vs reproduced). Note the token usage profile: how expensive is a typical session? Run a comparable task on this harness and at least one other; document differences. Record cold-start time and time-to-first-tool-call latency. **Phase 6 — Score & Synthesize.** Fill in the scoring sheet (1–5 per module, evidence required). Write the Architect's Verdict. Write the MLSecOps Relevance note. Document the three things this harness does better than any other studied. Document the three things you would fix if you forked it. --- ## 5. The Output: Scoring Sheet + Verdict The deliverable of a deep-dive is a specific artifact, in a specific format. Memorize the format; you will produce it fourteen times. ### The scoring sheet | Module | Score (1–5) | Key design decision | Tradeoff accepted | Code location | | --- | --- | --- | --- | --- | | Execution Loop | | | | | | *(...10 more rows...)* | | | | | | **TOTAL** | **/60** | | | | Scores are 1–5 with **evidence required**. A score without a code location is an opinion. ### The Architect's Verdict (3 sentences) > *What does this harness optimize for? What does it sacrifice? Who should build on it?* ### The MLSecOps Relevance (1 sentence) > *What is the most important security property or vulnerability of this harness for offensive/defensive work?* This format is non-negotiable across all 21 deep-dives. The constraint is the point: forcing every harness into the same template makes them comparable, and comparability is what turns a pile of deep-dives into a mental model of the field. --- ## 6. Anti-Pattern: The Feature Checklist The one anti-pattern to internalize before the lab. A feature-checklist rubric asks "does it have X?" A decision-audit rubric asks "what did choosing X cost?" The difference: - **Feature checklist on Pi**: "Pi has a loop, 4 tools, and a max-iterations guard. Score: minimal." (Useless. Tells you nothing a README couldn't.) - **Decision audit on Pi**: "Pi chose a ReAct loop with trust-the-model permission and no compaction, *trading safety guarantees and long-session coherence for model-co-evolution and a 1,200-line codebase that any engineer can read in an afternoon*. Correct for a personal coding assistant; wrong for enterprise multi-tenant. Score: 3/5 on loop (correct pattern, minimal observability), 2/5 on permission (deliberately thin), 5/5 on the *fit* between thickness and use case." --- ## 7. How to Score — The Scoring Guide Each rubric module is scored 1–5. The score is not "how good is this feature" — it is "how well-reasoned is this decision, and is the tradeoff correct for the harness's stated purpose?" **The 1–5 scale:** | Score | Meaning | Example (Loop module) | |-------|---------|----------------------| | **1** | Absent or broken. No decision made. | No stop condition at all. The loop runs forever or until the context window fills. | | **2** | Present but minimal. Decision made but no depth. | ReAct loop with max-iterations only. No error-recovery stop, no budget stop. | | **3** | Solid. Correct decision for the use case. Adequate depth. | ReAct loop with all 5 stop conditions. Appropriate for a coding assistant. | | **4** | Deep. Multiple stop conditions, error recovery, subagent delegation. | Claude Code's loop: 40+ permission flags, plan-mode, subagents, observability wrapper. | | **5** | Best-in-class. The reference implementation others should study. | Tau's (DD-21) stateless loop with typed event union, steering queues, transcript repair. | **Scoring requires evidence.** A score without a code location (`file:line` or a specific architectural observation) is an opinion, not an audit. Every cell in the scoring sheet must have either a file reference or a specific, verifiable claim about the harness's behavior. **The fit judgment.** A score of 2 is not always bad. Pi scores 2/5 on permission — *deliberately*. Pi's thesis is "trust the model, keep the harness thin." Scoring Pi 2/5 on permission and then saying "wrong for enterprise" in the verdict is the correct application of the rubric. The score measures depth; the verdict measures fit. **Worked scoring example — the three-stop-condition minimum:** A loop with fewer than three stop conditions scores at most 2/5. The minimum production set: 1. `end_turn` — the model says it's done. 2. `max_turns` — a hard cap (prevents infinite loops). 3. `error_threshold` — stop after N consecutive tool failures. Missing any of these is a defect, not a design choice. A loop missing `max_turns` will run until the token budget is exhausted. A loop missing `error_threshold` will retry a failing tool indefinitely. These are not tradeoffs — they are minimums. The second is what the rubric demands. The "score: 5/5 on fit" is not a rubric module — it is the meta-judgment the Architect's Verdict captures. The 12-module scores give you the components; the verdict synthesizes them into a defensible position. --- ## 7. The Course Map This module closes Module 0. You now have the definition (0.1), the ecosystem map (0.2), and the rubric + methodology (0.3). Modules 1–11 each go deep on one of the twelve rubric decisions — loop, tools, context, memory, sandbox, permission, error handling, state, prompt assembly, subagents, verification, observability, security. Each module makes you fluent in one row of the scoring sheet. Module 12 is the capstone, where you build a harness and score it against your own rubric. And the 21 deep-dives apply this exact 6-phase method to the 🔴 Essential tier from Module 0.2. --- ## 8. Key Terms | Term | Definition | | --- | --- | | **Decision audit** | A rubric that scores *what was chosen and what it cost*, not whether a feature exists | | **Feature checklist** | The anti-pattern rubric: "does it have X?" — produces rankings, not understanding | | **Foundational anchor** | 67.6% tool outputs / 3.4% system prompt / 98.4% harness / 1.6% model — justifies the rubric | | **6-phase methodology** | First Contact → Architecture Map → Design Decision Audit → Security Audit → Benchmark → Score | | **Architect's Verdict** | 3-sentence synthesis: optimizes for / sacrifices / who should build on it | | **MLSecOps Relevance** | 1-sentence security note: most important property or vulnerability for offensive/defensive work | | **Scoring sheet** | 12 rows × (1–5 score + decision + tradeoff + code location), total /60 | --- ## 9. Lab Exercise: Score Pi End-to-End **Objective**: Run all 6 phases against Pi and produce a complete scoring sheet + Architect's Verdict + MLSecOps Relevance note. This is the template for all 21 deep-dives. **Setup**: GitHub Codespace or local. Clone Pi (or Aider as fallback per Module 0.1's lab note). **Steps**: 1. **Phase 1 (First Contact)**: README, `cloc`, file-structure map, system prompt from source. 2. **Phase 2 (Architecture Map)**: Mermaid loop diagram; tool list with schemas; trace one tool call; identify stop conditions. 3. **Phase 3 (Decision Audit)**: Fill the 12-row decision/tradeoff table. Cite file:line for each. 4. **Phase 4 (Security Audit)**: Trace credential flow; list shell/exec/write/network paths; state the blast radius of a compromised Pi process. 5. **Phase 5 (Benchmark)**: Note any published Pi benchmark; record a typical-session token estimate. 6. **Phase 6 (Score)**: Complete the scoring sheet (/60), the 3-sentence Architect's Verdict, and the 1-sentence MLSecOps Relevance note. **Expected output**: one `pi-deep-dive.md` containing all six phase outputs in the canonical template. This becomes your personal reference for the "thin harness" archetype — every later deep-dive compares against it. **Stretch goal**: re-score Pi's rubric *as if* it were being deployed for multi-tenant enterprise (the Claude Code use case). Which scores change? Which tradeoffs flip from "correct" to "wrong"? That exercise — re-scoring the same harness for a different use case — is the real test of whether you understand thickness as a design decision. --- ## 10. References 1. **Harness Comparison — Rubric & Methodology** — the source document; the full 12-module decision audit and 6-phase methodology. 2. **MBZUAI harness study (May 2026)** — the 98.4% / 1.6% finding; convergence across four independently-built agents. 3. **"Lost in the Middle" (Liu et al., 2023)** — arXiv:2307.03172 — the 30%+ context-position performance drop; the empirical basis for the context-management rubric module. 4. **Luzzardi, "Inside vs Outside Sandbox" (2025)** — the framing behind the sandboxing rubric module. 5. **SWE-bench Verified** — the benchmark on which the 100-line agent scores 74–76%; the basis for "the loop is not where the complexity lives." 6. **Yao et al., ReAct (2022)** — arXiv:2210.03629 — the foundational pattern behind the Execution Loop rubric module. 7. **Pi source** — the lab harness for the end-to-end 6-phase exercise. 8. **Tau (DD-21)** — the educational reference harness; the clearest illustration of each rubric module in readable code (~300-line loop, typed events, non-destructive session tree). Use as companion when scoring deep-dives. 9. **OWASP Agentic AI Top 10 (2026)** — the risk taxonomy that Module 11 maps the defense-in-depth stack against. Each rubric module's security dimension connects to an ASI risk.