CS 319 · Meeting 5 of 10
Patterns for many agents — and the only thing standing between your loop and silent failure.
Agenda · 100 min
Reading: chapters 10–11
Core concept · ch 10
Supervisor (most common)
One orchestrator delegates to specialists and synthesizes. The council workflow is this. Use when a hub can hold the whole problem.
Hierarchical
Supervisors of supervisors — multi-level trees. Use when the org outgrows one hub; keep span-of-control small.
Group Chat
Peers debate in turns; a chairman synthesis closes it (Karpathy's council). Use for open-ended reasoning, not throughput.
Pipeline
Assembly line: spec → implement → review → deploy, artifacts flowing stage to stage. Use when stages have different tools/models.
Framework primitives you'll need regardless: state machine · message bus · tool allowlist per role.
The anti-pattern · and its cure
The 3000-token system prompt
"You are an architect AND backend dev AND reviewer AND writer AND DevOps…" Each role dilutes the others; every capability means a new contradiction; grading your own work is baked in.
The migration path
The tell: your "agent" has subheadings. Subheadings are specialists begging to be extracted.
Bridge · full treatment in Meeting 9
Mid-2026 discourse named the layer above loops graph engineering: the patterns on the previous slide are graph shapes — a supervisor is a hub, a pipeline is a chain, the message bus is the shared state.
The vocabulary adds one thing: edges as first-class design objects — conditional routing, fan-out, fan-in, drawn before they're coded.
Hold this thought for two weeks. Today's focus is the part every pattern shares and no pattern fixes for you: the verifier.
Core concept · ch 11
1 · Deterministic
Compilers, tests, linters, schema validators, exec-output matchers. Cheap, fast, binary. Always these first.
2 · LLM-as-judge
Scores against a rubric. Needs a calibration set (known good/bad) and a pinned model version — or your gate drifts silently.
3 · Human
Explicit gates at high-leverage moments only (Meeting 6). Humans are the most expensive checker; spend them on judgment, not patterns.
The rule this course runs on
The model that writes the code is too lenient when grading its own homework. Physically separate the maker from the checker — different agent, different context, ideally different model family.
Example · a verifier pipeline for one code-change loop
verify(out): 1 npm ci # determinstic: env reproduces 2 npm test -- --bail # deterministic: unit + e2e 3 eslint + tsc --strict # deterministic: static 4 validate api-diff vs api-contract.yaml # schema match 5 exec-match: run golden inputs, diff outputs 6 judge(model=pinned-v2, rubric.md, calibration/) # LLM layer 7 human gate: PR review # only after 1–6 pass order matters: fail fast at 1–5 costs cents; fail at 7 costs a human.
Most loops that "work in the demo" fail in production because steps 4–6 were missing — the maker checked its own homework at step 7.
Case study · part 5 of 10
Kirana's refund loop (M4) produced PR #212. Tests green. LLM self-review: "LGTM, 9/10." It almost merged.
The separate verifier node — different model, read-only, rubric pinned to the API contract — caught it: rounding applied before the 30-day window check on one path, so a day-31 refund of Rp 99,500 passed as day-30 Rp 100,000 in one edge case. Deterministic exec-match on golden inputs confirmed the mismatch.
The retro line that made it into the company playbook: "The maker said LGTM. The checker said show me."
Workshop · 30 minutes
Criterion to verify (from a real spec)
AC-3: "The weekly digest email contains every invoice status change from the past 7 days, grouped by customer, with correct totals."
Deliver (in pairs)
Trap to avoid: if your layer-1 check inspects the email's HTML, you are verifying the formatting, not the completeness. Verify against the source of truth.
Assignment A5 · due before Meeting 6
Task
For every acceptance criterion in your A3 spec: assign a layer (1/2/3), name the mechanism, and order the checks cheap→expensive. Physically separate maker and checker (different conversation, different model if you can).
Deliverables
Verifier pipeline (numbered, ordered, with costs) · run it on 3 real outputs: one good, one subtly bad, one garbage · calibration pair for your judge if you use layer 2.
Rubric
Layer assignment correctness 35% · ordering discipline 15% · the subtle-bad case: did the pipeline catch it? 50%.
Next: lab meeting — you scaffold and run your first complete loop, under human approval. Bring a laptop. Read ch 12–13.