CS 319 · Meeting 7 of 10
When the loop misbehaves at 3am, you will not fix it from the final output. You will need to rewind.
Agenda · 100 min
Reading: chapters 14–15 · Appendix B
Core concept · ch 14
The 3am question you can't answer
"The digest was wrong Tuesday. Why?" — with only the final output, you have a guess. The failure lived in what agent-2 saw at decision time, four steps earlier.
What you need instead
A time machine: what was in every agent's context window at every call, what each said, with what confidence, and what the next agent did with it.
Rule from the field: snapshot your assembled context before every model call, and store it next to the output. Storage is cheap; reconstruction is impossible.
Core concept · ch 14
1 · Structured logging
Who said what, with what confidence, in a parseable schema. Not prose. Not print lines.
2 · State snapshots
The assembled context per call, frozen at decision time. The unit of replay.
3 · Trace visualization
The graph of calls over time (LangSmith, OpenTelemetry, or a dot-file and patience). Where loops-within-loops become visible.
{"t":"2026-09-22T03:12:04Z","loop":"digest","run":88,"agent":"collector",
"call_id":"c4","tokens_in":4120,"tokens_out":380,
"inputs_hash":"a1f3","snapshot":"runs/88/c4.ctx.json",
"verdict":{"AC-1":"pass","AC-2":"pass","AC-3":"fail"},
"next":"retry(capped 2) | escalate"}
Core concept · ch 15
idempotent steps — a rerun must not double-charge, double-post, double-send
progress tracking — a crash at step 7/10 resumes, not restarts
graceful degradation — one source down → the digest ships with a hole and a note
cost + token budgets — hard caps per run and per day; a graph is many loops burning in parallel
context utilization ≤ 60–80% — headroom is a stability feature (ch 5, again)
model version pinning + prompt-prefix stability — upgrades are a deliberate act, never a surprise regression
And the artifact that makes all of it operational: the runbook — the page the on-call human reads at 3am. If it doesn't exist, your loop isn't production; it's a demo with uptime.
Artifact · 10 of the 75, chosen for this course
☐ every step idempotent
☐ budget caps wired and tested
☐ stagnation breaker armed
☐ context snapshots on every call
☐ skip logged differently from success
☐ model versions pinned, judge included
☐ landed-outcome column in the dashboard
☐ credentials scoped per agent, per role
☐ runbook written and dry-run by a stranger
☐ a kill switch a human can find at 3am
Items 5 and 7 come from fleet-audit practice (Meeting 10): a loop that skipped must never log ok — that single distinction hid a dead loop for a week in the documented case.
Case study · part 7 of 10
Kirana's digest loop had been stable at ~9k tokens/run for three weeks. Then Tuesday: 38k. Output looked fine. Without snapshots this is a shrug.
With them: diff runs/88/c4.ctx.json against runs/81/c4.ctx.json. The collector agent's context had grown a raw HTML dump — a "small improvement" a developer had made to the fetcher on Monday. One sub-agent boundary (ch 5) had been quietly removed, and the swamp moved upstream into every run.
Fix: restore the distillation step + a budget assertion in the verifier (fail the run if tokens_in > 12k). The token graph went flat. The retro wrote itself: "the snapshot diff was the whole investigation."
Workshop · 25 minutes
Your A6 loop, one page, for a stranger
Test: hand it to the pair next to you. They have 90 seconds to find the kill switch without asking you a question. If they can't, revise.
Assignment A7 · due before Meeting 8
Task
Add structured logging and per-call context snapshots to your A6 loop. Induce one failure (feed it a bad input, kill a dependency, shrink a budget) and replay the failure from logs only — no watching it live. Then finalize the runbook.
Deliverables
Three log lines from the replay + one snapshot diff with annotation ("here is where it went wrong") · final runbook · the 10-item checklist, honestly ticked.
Rubric
Replay-from-logs actually works 40% · snapshot diff annotation 30% · runbook stranger-test 30%.
Next week: the human layer — coaching, codifying, and the anti-pattern catalog. Read ch 16–17.