Field Notes

What breaks in a multi-agent pipeline (and how we bounded it)

July 18, 2026 · 7 min read

Multi-agent demos are easy. Multi-agent systems that survive real runs, flaky networks, model quotas and hostile input are a different discipline. While building Proposal Architect for the Google for Startups AI Agents Challenge 2026, most of our engineering time went not into making agents talk, but into bounding what happens when they fail.

These are the field notes: six things that broke, why they broke, and the pattern that fixed each one.

1. The call that never returns

Symptom: a pipeline run freezes at one stage and sits there forever.

Root cause: a model call with no deadline. Under load, an inference endpoint will occasionally neither answer nor fail: it just hangs, and an unbounded await turns one slow call into a dead pipeline.

Pattern: every model call gets a hard timeout, and every run gets a hard overall deadline. Recovery is only possible when failure is detectable, and failure is only detectable when waiting has a limit.

2. Retrying everything to fix one thing

Symptom: the last stage fails, and the naive retry re-runs the entire pipeline: slow, expensive, and it can produce a different proposal than the one that almost succeeded.

Root cause: retry logic that treats the pipeline as one atomic unit.

Pattern: checkpoint state between agents and retry only the failed stage. When our composer failed, we re-ran the composer against the already-computed upstream state; the architecture, the estimates and the price stayed exactly as approved.

3. Preview-model quotas run out at the worst time

Symptom: the pipeline dies mid-demo with quota errors.

Root cause: preview-tier models come with tight, shared quotas that behave nothing like generally-available models under repeated runs.

Pattern: route load-bearing paths to generally-available models and reserve preview or premium tiers for the one step where quality visibly compounds (in our case, the architect agent). Tier routing is a reliability decision as much as a cost one.

4. The critic that criticizes from memory

Symptom: the adversarial critic rejects valid drafts with claims about model capabilities that are simply out of date.

Root cause: a critic grounded on training memory hallucinates the present.

Pattern: ground the critic on live facts (in our case, a current model catalog), so its objections reference reality instead of recollection.

5. The brief is hostile input

Symptom: none, and that is the problem: you rarely see the injection that worked.

Root cause: a client brief is user-controlled text that flows into every agent's context. That makes it an injection surface.

Pattern: treat the brief as untrusted input. Harden the system prompts, and validate everything the model authors at the boundary where it becomes consequential. Our deterministic pricing gate rejects malformed numbers regardless of why they are malformed: a hallucination and an attack get the same answer.

6. The phone locked and the run "disappeared"

Symptom: on mobile, a user reloads mid-run and loses the pipeline they were watching.

Root cause: run state living in the browser session.

Pattern: the run belongs to the server; the client only observes it. After a reload, the UI reattaches to the in-flight run instead of starting a new one.

The meta-pattern

Every fix above is the same fix wearing different clothes: define the boundary, bound the wait, keep the state, ground the facts. Agent frameworks give you the conversation between models. The engineering is everything around it.