Module / Section: 0 / 0.3 · Duration: ~15 minutes · Target: ~2,300 words
Format: Verbatim transcript. Cues [SLIDE N] map to 03-slide-deck.html.
[SLIDE 1 — Title]
This is Module Zero, Section Three: The Rubric. It is the last section of the landscape module, and it is the one that turns the rest of the course from a tour into a discipline. Modules 0.1 and 0.2 gave you the definition and the ecosystem map. This module gives you the method. By the end you will have a twelve-module scoring rubric, a six-phase deep-dive methodology, and the exact output template you will produce for every one of the twenty-one deep-dives later in the course.
[SLIDE 2 — The foundational anchor]
Before the rubric, one empirical anchor that justifies the whole framework. Here is what the model actually sees in context in a typical production session. Tool outputs are about sixty-seven point six percent. Tool definitions are another ten point seven. History and everything else is around eighteen. The system prompt — the thing most teams spend most of their time polishing — is three point four percent. And the model's own decision logic is that one point six percent from Module 0.1.
Hold those two bold numbers. If you have been iterating on your system prompt while ignoring how your tool outputs are formatted, you have been optimizing three percent of context while neglecting sixty-eight percent. That is the single most common senior-engineer mistake in this entire domain. Modules Two and Three exist to fix it. Everything in this rubric is downstream of that asymmetry.
[SLIDE 3 — The paradox and its resolution]
Now a paradox, and its resolution, because the resolution is the load-bearing claim of this module. A roughly hundred-line Python agent scores seventy-four to seventy-six percent on SWE-bench Verified — only four to six points behind state-of-the-art systems built by well-funded teams. And the MBZUAI study found about ninety-eight point four percent of a production harness is infrastructure and only one point six percent is AI decision logic.
So is the loop all that matters? No. Here is the precise resolution. The execution loop is not where the complexity lives. World-class teams invest six to twelve months building the infrastructure around that loop — context management, tool design, memory, verification, observability, security. The last four to six SWE-bench points, and more importantly the reliability, security, and governance properties that benchmarks do not measure, live entirely in the ninety-eight point four percent.
Now hear the two failure modes, because they are symmetric. A learner who concludes "the loop is all that matters" because the hundred-line agent is competitive has misunderstood. But a learner who concludes "the loop doesn't matter" has misunderstood worse. The loop is necessary and cheap. The ninety-eight point four percent around it is where you earn your keep. The rubric scores that ninety-eight point four percent.
[SLIDE 4 — Decision audit, not feature checklist]
Most rubrics in software are feature checklists. Does it have a sandbox? Does it have compaction? Does it have multi-session? Feature checklists produce a ranking. They do not produce understanding.
This rubric is different. It is a decision audit. Each of the twelve modules asks three questions. What pattern did they choose? What did they give up? Where in the code is the evidence? The tell that you are doing it wrong: if your rubric output for a harness could be generated by reading its README's feature list, you have not audited a decision. You have transcribed marketing.
[SLIDE 5 — "Has a sandbox" vs a real decision]
Make it concrete. Here is a feature-checklist output on Pi. "Pi has a loop, four tools, and a max-iterations guard." Useless. Tells you nothing the README couldn't.
Here is a decision-audit output on the same harness. "Pi chose ReAct with trust-the-model permission and no compaction, trading safety guarantees and long-session coherence for model-co-evolution and a twelve-hundred-line codebase any engineer can read in an afternoon. Correct for a personal coding assistant; wrong for enterprise multi-tenant."
The difference is everything. The second names a pattern, names the tradeoff, and states the use-case fit. That is what the rubric scores. Notice also why Module 0.2's NemoClaw can inherit ninety percent of OpenClaw's code and still be a distinct, scorable harness: its entire score is in the governance decision, not the inherited features.
[SLIDE 6 — The twelve modules]
The rubric has twelve modules. Each is a "where does the complexity live" question. The first six go deep on a layer; the next six handle cross-cutting concerns. Execution loop — ReAct versus plan-then-execute versus graph-based versus dumb-loop versus conversation-driven. Tool design — static versus dynamic versus MCP, schema-first versus free-text, and the consequential choice of tool count. Context management — compaction, observation masking, JIT retrieval, structured note-taking, subagent delegation. Memory — in-context, working files, semantic store, episodic log, structured DB. Sandboxing — the fundamental inside-versus-outside split we previewed in Module 0.2. Permission — auto-approve, deny-by-default, risk-tiered, plan-mode, per-flag, HITL checkpoints. Error handling — the four-category taxonomy: transient, LLM-recoverable, user-fixable, fatal. State and checkpointing — git, LangGraph super-steps, external state file, DB-backed, or none. Prompt assembly — system prompt size, tool injection location, memory placement, truncation strategy. Subagent orchestration — agents-as-tools, handoffs, fork, teammate, worktree. Verification — computed, visual, model-judged, human-in-loop. And observability — structured logs, traces, token accounting, replay, metrics export, session diffing.
You do not memorize the patterns. You learn to recognize them and to name the tradeoff.
[SLIDE 7 — The 6-phase methodology]
The rubric tells you what to score. The methodology tells you how to score it, reproducibly, on any harness. Six phases.
Phase one, First Contact. Read the README top to bottom; note what is not explained. Run cloc, record lines by language. Map the file structure from entry point outwards. Find and read the system prompt in full source, not docs.
Phase two, Architecture Map. Draw the execution loop as a Mermaid diagram. List every tool with its schema. Trace a single tool call end to end. Identify all stop conditions.
Phase three, Design Decision Audit. For each of the twelve modules: what pattern, and why? For each pattern, what did they give up? Find three decisions you agree with; find three you would make differently.
Phase four, Security Audit. Trace credential flow. Identify every shell, exec, write, and network path. Test whether a tool output can inject an instruction that changes behavior. State the blast radius of a compromised agent process.
Phase five, Benchmark and Performance. Record published scores and whether they are self-reported or reproduced. Note the token profile. Run a comparable task on this harness and one other; document the difference.
Phase six, Score and Synthesize. Fill the scoring sheet — one to five per module, evidence required. Write the Architect's Verdict. Write the MLSecOps Relevance note. Document three things this harness does better than any other studied. Document three things you would fix if you forked it.
[SLIDE 8 — The deliverable format]
The deliverable of a deep-dive is a specific artifact, in a specific format. Memorize it; you will produce it fourteen times. A scoring sheet: twelve rows, each with a one-to-five score, the key design decision, the tradeoff accepted, and the code location. Total out of sixty. Scores are one to five with evidence required — a score without a code location is an opinion. Then the Architect's Verdict, exactly three sentences: what does this harness optimize for, what does it sacrifice, who should build on it. Then the MLSecOps Relevance note, exactly one sentence: the most important security property or vulnerability for offensive or defensive work.
The format constraint is the point. Forcing every harness into the same template makes them comparable. Comparability is what turns a pile of deep-dives into a mental model of the field.
[SLIDE 9 — The 6 phases as a workflow]
Per the course's visual stack, n8n first. The six-phase methodology is a process, and processes are what n8n visualizes best. Trigger picks a harness. Phase one runs cloc and structure. Phase two greps for the loop and tools. Phase three gathers decision evidence. Phase four gathers security evidence. Phase five pulls benchmark data. Phase six emits the templates — the scoring sheet, the verdict, the MLSecOps note.
One thing to be precise about: the workflow does not score for you. A human, or an LLM judge, fills the templates from the grep evidence the earlier phases produced. What the workflow does is operationalize the methodology so no phase gets skipped. The number of times a deep-dive goes wrong because someone skipped phase four — the security audit — and shipped a verdict that missed the harness's central vulnerability is exactly why this workflow exists.
[SLIDE 10 — Takeaways]
Five things to leave with.
One: the anchor. Sixty-seven point six tool outputs, three point four system prompt, ninety-eight point four harness, one point six model.
Two: the rubric is a decision audit, not a feature checklist. Score what was chosen and what it cost. If your output could come from a README, you transcribed marketing, not audited a decision.
Three: six phases, same for all twenty-one deep-dives. First Contact, Architecture Map, Decision Audit, Security Audit, Benchmark, Score and Synthesize.
Four: the deliverable is out of sixty, plus a three-sentence Architect's Verdict, plus a one-sentence MLSecOps Relevance note.
Five: the format constraint is the point. Comparability turns deep-dives into a mental model.
Module Zero is now complete. You have the definition, the ecosystem, and the method. Next, Module One — the execution loop, in depth. The first of the twelve rubric decisions.
End of Module 0.3 teaching script. Approximate word count: 2,290. Approximate runtime at 140 wpm: 16 minutes.
# Teaching Script — Module 0.3: The Rubric **Module / Section**: 0 / 0.3 · **Duration**: ~15 minutes · **Target**: ~2,300 words **Format**: Verbatim transcript. Cues `[SLIDE N]` map to `03-slide-deck.html`. --- [SLIDE 1 — Title] This is Module Zero, Section Three: The Rubric. It is the last section of the landscape module, and it is the one that turns the rest of the course from a tour into a discipline. Modules 0.1 and 0.2 gave you the definition and the ecosystem map. This module gives you the method. By the end you will have a twelve-module scoring rubric, a six-phase deep-dive methodology, and the exact output template you will produce for every one of the twenty-one deep-dives later in the course. [SLIDE 2 — The foundational anchor] Before the rubric, one empirical anchor that justifies the whole framework. Here is what the model actually sees in context in a typical production session. Tool outputs are about sixty-seven point six percent. Tool definitions are another ten point seven. History and everything else is around eighteen. The system prompt — the thing most teams spend most of their time polishing — is three point four percent. And the model's own decision logic is that one point six percent from Module 0.1. Hold those two bold numbers. If you have been iterating on your system prompt while ignoring how your tool outputs are formatted, you have been optimizing three percent of context while neglecting sixty-eight percent. That is the single most common senior-engineer mistake in this entire domain. Modules Two and Three exist to fix it. Everything in this rubric is downstream of that asymmetry. [SLIDE 3 — The paradox and its resolution] Now a paradox, and its resolution, because the resolution is the load-bearing claim of this module. A roughly hundred-line Python agent scores seventy-four to seventy-six percent on SWE-bench Verified — only four to six points behind state-of-the-art systems built by well-funded teams. And the MBZUAI study found about ninety-eight point four percent of a production harness is infrastructure and only one point six percent is AI decision logic. So is the loop all that matters? No. Here is the precise resolution. The execution loop is *not* where the complexity lives. World-class teams invest six to twelve months building the infrastructure *around* that loop — context management, tool design, memory, verification, observability, security. The last four to six SWE-bench points, and more importantly the reliability, security, and governance properties that benchmarks do not measure, live entirely in the ninety-eight point four percent. Now hear the two failure modes, because they are symmetric. A learner who concludes "the loop is all that matters" because the hundred-line agent is competitive has misunderstood. But a learner who concludes "the loop doesn't matter" has misunderstood *worse*. The loop is necessary and cheap. The ninety-eight point four percent around it is where you earn your keep. The rubric scores that ninety-eight point four percent. [SLIDE 4 — Decision audit, not feature checklist] Most rubrics in software are feature checklists. Does it have a sandbox? Does it have compaction? Does it have multi-session? Feature checklists produce a ranking. They do not produce understanding. This rubric is different. It is a *decision audit*. Each of the twelve modules asks three questions. What pattern did they choose? What did they give up? Where in the code is the evidence? The tell that you are doing it wrong: if your rubric output for a harness could be generated by reading its README's feature list, you have not audited a decision. You have transcribed marketing. [SLIDE 5 — "Has a sandbox" vs a real decision] Make it concrete. Here is a feature-checklist output on Pi. "Pi has a loop, four tools, and a max-iterations guard." Useless. Tells you nothing the README couldn't. Here is a decision-audit output on the same harness. "Pi chose ReAct with trust-the-model permission and no compaction, *trading safety guarantees and long-session coherence for model-co-evolution and a twelve-hundred-line codebase any engineer can read in an afternoon*. Correct for a personal coding assistant; wrong for enterprise multi-tenant." The difference is everything. The second names a pattern, names the tradeoff, and states the use-case fit. That is what the rubric scores. Notice also why Module 0.2's NemoClaw can inherit ninety percent of OpenClaw's code and still be a distinct, scorable harness: its entire score is in the governance decision, not the inherited features. [SLIDE 6 — The twelve modules] The rubric has twelve modules. Each is a "where does the complexity live" question. The first six go deep on a layer; the next six handle cross-cutting concerns. Execution loop — ReAct versus plan-then-execute versus graph-based versus dumb-loop versus conversation-driven. Tool design — static versus dynamic versus MCP, schema-first versus free-text, and the consequential choice of tool count. Context management — compaction, observation masking, JIT retrieval, structured note-taking, subagent delegation. Memory — in-context, working files, semantic store, episodic log, structured DB. Sandboxing — the fundamental inside-versus-outside split we previewed in Module 0.2. Permission — auto-approve, deny-by-default, risk-tiered, plan-mode, per-flag, HITL checkpoints. Error handling — the four-category taxonomy: transient, LLM-recoverable, user-fixable, fatal. State and checkpointing — git, LangGraph super-steps, external state file, DB-backed, or none. Prompt assembly — system prompt size, tool injection location, memory placement, truncation strategy. Subagent orchestration — agents-as-tools, handoffs, fork, teammate, worktree. Verification — computed, visual, model-judged, human-in-loop. And observability — structured logs, traces, token accounting, replay, metrics export, session diffing. You do not memorize the patterns. You learn to *recognize* them and to *name the tradeoff*. [SLIDE 7 — The 6-phase methodology] The rubric tells you what to score. The methodology tells you how to score it, reproducibly, on any harness. Six phases. Phase one, First Contact. Read the README top to bottom; note what is *not* explained. Run cloc, record lines by language. Map the file structure from entry point outwards. Find and read the system prompt in full source, not docs. Phase two, Architecture Map. Draw the execution loop as a Mermaid diagram. List every tool with its schema. Trace a single tool call end to end. Identify all stop conditions. Phase three, Design Decision Audit. For each of the twelve modules: what pattern, and why? For each pattern, what did they give up? Find three decisions you agree with; find three you would make differently. Phase four, Security Audit. Trace credential flow. Identify every shell, exec, write, and network path. Test whether a tool output can inject an instruction that changes behavior. State the blast radius of a compromised agent process. Phase five, Benchmark and Performance. Record published scores and whether they are self-reported or reproduced. Note the token profile. Run a comparable task on this harness and one other; document the difference. Phase six, Score and Synthesize. Fill the scoring sheet — one to five per module, evidence required. Write the Architect's Verdict. Write the MLSecOps Relevance note. Document three things this harness does better than any other studied. Document three things you would fix if you forked it. [SLIDE 8 — The deliverable format] The deliverable of a deep-dive is a specific artifact, in a specific format. Memorize it; you will produce it fourteen times. A scoring sheet: twelve rows, each with a one-to-five score, the key design decision, the tradeoff accepted, and the code location. Total out of sixty. Scores are one to five with *evidence required* — a score without a code location is an opinion. Then the Architect's Verdict, exactly three sentences: what does this harness optimize for, what does it sacrifice, who should build on it. Then the MLSecOps Relevance note, exactly one sentence: the most important security property or vulnerability for offensive or defensive work. The format constraint is the point. Forcing every harness into the same template makes them comparable. Comparability is what turns a pile of deep-dives into a mental model of the field. [SLIDE 9 — The 6 phases as a workflow] Per the course's visual stack, n8n first. The six-phase methodology is a *process*, and processes are what n8n visualizes best. Trigger picks a harness. Phase one runs cloc and structure. Phase two greps for the loop and tools. Phase three gathers decision evidence. Phase four gathers security evidence. Phase five pulls benchmark data. Phase six emits the templates — the scoring sheet, the verdict, the MLSecOps note. One thing to be precise about: the workflow does not score *for* you. A human, or an LLM judge, fills the templates from the grep evidence the earlier phases produced. What the workflow does is operationalize the methodology so no phase gets skipped. The number of times a deep-dive goes wrong because someone skipped phase four — the security audit — and shipped a verdict that missed the harness's central vulnerability is exactly why this workflow exists. [SLIDE 10 — Takeaways] Five things to leave with. One: the anchor. Sixty-seven point six tool outputs, three point four system prompt, ninety-eight point four harness, one point six model. Two: the rubric is a decision audit, not a feature checklist. Score what was chosen and what it cost. If your output could come from a README, you transcribed marketing, not audited a decision. Three: six phases, same for all twenty-one deep-dives. First Contact, Architecture Map, Decision Audit, Security Audit, Benchmark, Score and Synthesize. Four: the deliverable is out of sixty, plus a three-sentence Architect's Verdict, plus a one-sentence MLSecOps Relevance note. Five: the format constraint is the point. Comparability turns deep-dives into a mental model. Module Zero is now complete. You have the definition, the ecosystem, and the method. Next, Module One — the execution loop, in depth. The first of the twelve rubric decisions. --- *End of Module 0.3 teaching script. Approximate word count: 2,290. Approximate runtime at 140 wpm: 16 minutes.*