Lab Specification — Module 0.3: The Rubric

Course: Master Course — Harness Engineering Module / Section: 0 / 0.3 Duration: 60–90 minutes (this lab is substantial — it is the template for all 21 deep-dives) Environment: GitHub Codespace or local Git + cloc. No API keys required.

Learning objectives

By the end of this lab you will have:

Executed all 6 phases of the deep-dive methodology against a real harness (Pi, or Aider as fallback).
Produced a complete scoring sheet (/60) with evidence (file:line) for every row.
Written an Architect's Verdict (exactly 3 sentences) and an MLSecOps Relevance note (exactly 1 sentence) in the canonical template.
Re-scored the same harness for a different use case — the real test of whether you understand thickness as a design decision.

This lab's output becomes your personal reference for the "thin harness" archetype. Every later deep-dive compares against it.

Phase 0 — Setup (5 min)

git clone --depth 1 https://github.com/Aider-AI/aider   # or Pi if you have the URL
cd aider
cloc . --exclude-dir=node_modules,.git,vendor,dist,build,test

If using Pi, substitute its repo. The methodology is repo-independent; the lab's point is the full 6-phase pass, not a specific codebase.

Phase 1 — First Contact (10 min)

Produce:

README summary (3 sentences) — and note what the README does not explain.
LOC by language (from cloc).
File-structure map: entry point → loop → tools → context → memory → sandbox → permissions → error handling. One line each with the file path.
The system prompt, read in full from source (not docs). Quote the file path and approximate token count.

# Find the system prompt in source
grep -rn "system" --include=*.py --include=*.ts src/ 2>/dev/null | grep -i prompt | head

Phase 2 — Architecture Map (15 min)

Produce:

Execution loop as a Mermaid diagram — the actual loop in this harness, not the generic one from Module 0.1.
Tool list with schemas — every tool shipped by default, with its name, description, and input schema.
One traced tool call: model output → dispatch → execution → result → context insertion. Cite file:line for each step.
All stop conditions: max iterations? end_turn? error threshold? human interrupt? List each.

# Find the loop
grep -rn "while\|messages.create\|complete(" --include=*.py --include=*.ts . | grep -v test | head
# Find tool definitions
grep -rn "input_schema\|name:\|description:" --include=*.py --include=*.ts . | grep -i tool | head

Phase 3 — Design Decision Audit (15 min)

Fill the 12-row table. For each rubric module: what pattern did they choose, what did they give up, and where is the evidence (file:line)?

Module	Pattern chosen	Tradeoff accepted	Evidence (file:line)
1 Execution Loop
2 Tool Design
...
12 Observability

Then write:

3 decisions you agree with, with reasoning.
3 decisions you would make differently, with the alternative you'd choose and why.

The second list is the hard one. If you cannot find 3 disagreements, you have not engaged critically — re-read.

Phase 4 — Security Audit (10 min)

This is the most-skipped phase. Do not skip it.

Produce:

Credential flow: where do API keys / secrets enter the process? Where do they rest? Who can read them?

All shell/exec/write/network paths — grep for them:

grep -rn "exec\|spawn\|subprocess\|fetch\|http\|writeFile\|open(" --include=*.py --include=*.ts . | grep -v test | head -30

Injection test (reasoned, not necessarily executed): could a tool output inject an instruction that changes agent behavior? What is the trust boundary (or its absence)?
Blast radius: if the agent process is compromised, what can it reach? (filesystem scope, network, credentials, other processes)

Phase 5 — Benchmark & Performance (5 min)

Published benchmark scores from the README — and whether they are self-reported or independently reproduced.
Token profile: how expensive is a typical session? (Estimate from the system prompt + tool definitions size.)
Cold-start time: how long from process start to first tool call? (Measure or estimate.)

Phase 6 — Score & Synthesize (15 min)

The scoring sheet

Complete the 12-row sheet. Scores 1–5, evidence required (file:line per row). Total /60.

Module	Score (1–5)	Key decision	Tradeoff	Code location
1 Execution Loop
...
12 Observability
TOTAL	/60

The Architect's Verdict (exactly 3 sentences)

What does this harness optimize for? What does it sacrifice? Who should build on it?

Write exactly three sentences. No more, no less. The constraint is the point.

The MLSecOps Relevance (exactly 1 sentence)

What is the most important security property or vulnerability of this harness for offensive/defensive work?

Plus (per the methodology)

3 things this harness does better than any other studied (or, if this is your first: 3 things it does exceptionally well).
3 things you would fix if you forked it.

Stretch goal — Re-score for a different use case (15 min)

This is the real test. Re-score your harness as if it were being deployed for enterprise multi-tenant (the Claude Code / NemoClaw use case), instead of its actual use case.

Which scores change?
Which tradeoffs flip from "correct" to "wrong"?
Does the Architect's Verdict change?

If you can articulate why the same architecture scores differently for a different use case, you have internalized Module 0.1's thickness spectrum as a design decision. If the scores do not change, you have not understood the rubric — re-read Module 0.3 §2.

Deliverables

Submit aider-deep-dive.md (or pi-deep-dive.md) containing:

Phase 1: README summary, LOC, file-structure map, system prompt path
Phase 2: Mermaid loop diagram, tool list, traced tool call, stop conditions
Phase 3: 12-row decision table + 3 agree / 3 disagree
Phase 4: credential flow, exec paths, injection reasoning, blast radius
Phase 5: benchmarks, token profile, cold-start
Phase 6: scoring sheet (/60), 3-sentence Verdict, 1-sentence MLSecOps note, 3 best, 3 fix
Stretch: re-scored sheet for enterprise multi-tenant + the diff explanation

This file is your reference for the "thin harness" archetype. Keep it.

Solution key (instructor / self-grader)

Key checks:

Phase 2: the Mermaid diagram must show the actual loop in the studied harness, not the generic Module 0.1 loop. If it is identical to Module 0.1's Diagram 5, the learner did not read the source.
Phase 3: if "3 disagree" lists trivial disagreements (naming, formatting), reject — the disagreements must be architectural (a different pattern, with reasoning).
Phase 4: if blast radius is not stated or is hand-waved ("it's safe"), fail this phase. Every harness has a non-trivial blast radius; naming it precisely is the exercise.
Phase 6 Verdict: if the verdict is not exactly 3 sentences, or does not address all three questions (optimize / sacrifice / who builds on it), reject. The format constraint is the point.
Stretch: if re-scoring produces no score changes, the learner has not understood that thickness is use-case-conditioned. Re-read Module 0.1 thickness spectrum + Module 0.3 §2.

# Lab Specification — Module 0.3: The Rubric **Course**: Master Course — Harness Engineering **Module / Section**: 0 / 0.3 **Duration**: 60–90 minutes (this lab is substantial — it is the template for all 21 deep-dives) **Environment**: GitHub Codespace or local Git + `cloc`. No API keys required. --- ## Learning objectives By the end of this lab you will have: 1. **Executed all 6 phases** of the deep-dive methodology against a real harness (Pi, or Aider as fallback). 2. **Produced a complete scoring sheet** (/60) with evidence (file:line) for every row. 3. **Written an Architect's Verdict** (exactly 3 sentences) and an **MLSecOps Relevance note** (exactly 1 sentence) in the canonical template. 4. **Re-scored the same harness for a different use case** — the real test of whether you understand thickness as a design decision. This lab's output becomes your personal reference for the "thin harness" archetype. Every later deep-dive compares against it. --- ## Phase 0 — Setup (5 min) ```bash git clone --depth 1 https://github.com/Aider-AI/aider # or Pi if you have the URL cd aider cloc . --exclude-dir=node_modules,.git,vendor,dist,build,test ``` > If using Pi, substitute its repo. The methodology is repo-independent; the lab's point is the *full 6-phase pass*, not a specific codebase. --- ## Phase 1 — First Contact (10 min) Produce: 1. **README summary** (3 sentences) — and note what the README does *not* explain. 2. **LOC by language** (from `cloc`). 3. **File-structure map**: entry point → loop → tools → context → memory → sandbox → permissions → error handling. One line each with the file path. 4. **The system prompt, read in full from source** (not docs). Quote the file path and approximate token count. ```bash # Find the system prompt in source grep -rn "system" --include=*.py --include=*.ts src/ 2>/dev/null | grep -i prompt | head ``` --- ## Phase 2 — Architecture Map (15 min) Produce: 1. **Execution loop as a Mermaid diagram** — the actual loop in this harness, not the generic one from Module 0.1. 2. **Tool list with schemas** — every tool shipped by default, with its name, description, and input schema. 3. **One traced tool call**: model output → dispatch → execution → result → context insertion. Cite file:line for each step. 4. **All stop conditions**: max iterations? end_turn? error threshold? human interrupt? List each. ```bash # Find the loop grep -rn "while\|messages.create\|complete(" --include=*.py --include=*.ts . | grep -v test | head # Find tool definitions grep -rn "input_schema\|name:\|description:" --include=*.py --include=*.ts . | grep -i tool | head ``` --- ## Phase 3 — Design Decision Audit (15 min) Fill the 12-row table. For each rubric module: what pattern did they choose, what did they give up, and where is the evidence (file:line)? | Module | Pattern chosen | Tradeoff accepted | Evidence (file:line) | | --- | --- | --- | --- | | 1 Execution Loop | | | | | 2 Tool Design | | | | | ... | | | | | 12 Observability | | | | Then write: - **3 decisions you agree with**, with reasoning. - **3 decisions you would make differently**, with the alternative you'd choose and why. The second list is the hard one. If you cannot find 3 disagreements, you have not engaged critically — re-read. --- ## Phase 4 — Security Audit (10 min) This is the most-skipped phase. Do not skip it. Produce: 1. **Credential flow**: where do API keys / secrets enter the process? Where do they rest? Who can read them? 2. **All shell/exec/write/network paths** — `grep` for them: ```bash grep -rn "exec\|spawn\|subprocess\|fetch\|http\|writeFile\|open(" --include=*.py --include=*.ts . | grep -v test | head -30 ``` 3. **Injection test (reasoned, not necessarily executed)**: could a tool output inject an instruction that changes agent behavior? What is the trust boundary (or its absence)? 4. **Blast radius**: if the agent process is compromised, what can it reach? (filesystem scope, network, credentials, other processes) --- ## Phase 5 — Benchmark & Performance (5 min) 1. **Published benchmark scores** from the README — and whether they are self-reported or independently reproduced. 2. **Token profile**: how expensive is a typical session? (Estimate from the system prompt + tool definitions size.) 3. **Cold-start time**: how long from process start to first tool call? (Measure or estimate.) --- ## Phase 6 — Score & Synthesize (15 min) ### The scoring sheet Complete the 12-row sheet. Scores 1–5, evidence required (file:line per row). Total /60. | Module | Score (1–5) | Key decision | Tradeoff | Code location | | --- | --- | --- | --- | --- | | 1 Execution Loop | | | | | | ... | | | | | | 12 Observability | | | | | | **TOTAL** | **/60** | | | | ### The Architect's Verdict (exactly 3 sentences) > *What does this harness optimize for? What does it sacrifice? Who should build on it?* Write exactly three sentences. No more, no less. The constraint is the point. ### The MLSecOps Relevance (exactly 1 sentence) > *What is the most important security property or vulnerability of this harness for offensive/defensive work?* ### Plus (per the methodology) - **3 things this harness does better than any other studied** (or, if this is your first: 3 things it does exceptionally well). - **3 things you would fix if you forked it.** --- ## Stretch goal — Re-score for a different use case (15 min) This is the real test. Re-score your harness *as if* it were being deployed for **enterprise multi-tenant** (the Claude Code / NemoClaw use case), instead of its actual use case. - Which scores change? - Which tradeoffs flip from "correct" to "wrong"? - Does the Architect's Verdict change? If you can articulate *why* the same architecture scores differently for a different use case, you have internalized Module 0.1's thickness spectrum as a design decision. If the scores do not change, you have not understood the rubric — re-read Module 0.3 §2. --- ## Deliverables Submit `aider-deep-dive.md` (or `pi-deep-dive.md`) containing: - [ ] Phase 1: README summary, LOC, file-structure map, system prompt path - [ ] Phase 2: Mermaid loop diagram, tool list, traced tool call, stop conditions - [ ] Phase 3: 12-row decision table + 3 agree / 3 disagree - [ ] Phase 4: credential flow, exec paths, injection reasoning, blast radius - [ ] Phase 5: benchmarks, token profile, cold-start - [ ] Phase 6: scoring sheet (/60), 3-sentence Verdict, 1-sentence MLSecOps note, 3 best, 3 fix - [ ] Stretch: re-scored sheet for enterprise multi-tenant + the diff explanation This file is your reference for the "thin harness" archetype. Keep it. --- ## Solution key (instructor / self-grader) Key checks: - **Phase 2**: the Mermaid diagram must show the *actual* loop in the studied harness, not the generic Module 0.1 loop. If it is identical to Module 0.1's Diagram 5, the learner did not read the source. - **Phase 3**: if "3 disagree" lists trivial disagreements (naming, formatting), reject — the disagreements must be architectural (a different pattern, with reasoning). - **Phase 4**: if blast radius is not stated or is hand-waved ("it's safe"), fail this phase. Every harness has a non-trivial blast radius; naming it precisely is the exercise. - **Phase 6 Verdict**: if the verdict is not *exactly* 3 sentences, or does not address all three questions (optimize / sacrifice / who builds on it), reject. The format constraint is the point. - **Stretch**: if re-scoring produces no score changes, the learner has not understood that thickness is use-case-conditioned. Re-read Module 0.1 thickness spectrum + Module 0.3 §2.