Retry nodes, fallback edges: error handling as graph topology

Our due-diligence pipeline talks to nine-ish external services, and exactly one of them shaped the architecture of everything else: a company-data vendor whose API times out on somewhere between 1 and 3% of calls, worse on Mondays. I graphed it. I still can't explain Mondays.

In April, before the fan-out went parallel and a run still took the better part of an hour, one of those timeouts landed forty minutes in and killed the entire run. Nothing was wrong with the other eight sources, or the drafts already written: one HTTP timeout, one dead report. It happened often enough to earn a nickname on the client's side, which is never the sign you want.

the swallowing mistake

My first fix made it worse, and I want to record that honestly.

I wrapped every node body in a catch-all try/except that logged the error and returned an empty dict. Runs stopped dying. Reports started shipping with silent holes where sections should have been, and because the compile step handled missing data gracefully, nothing flagged it. It took an analyst asking why the sanctions section had been blank two days running for me to notice what I'd done.

Crashing loudly was better than that. A silent gap is a lie with a timestamp on it.

So the rework had one principle: failures may not stop the run, and they may not hide. Everything below follows from those two clauses.

retries where the flake lives

LangGraph lets you attach a retry policy to the node itself, which pulled the first pile of spaghetti out of my node bodies:

builder.add_node(
    "enrich",
    enrich,
    retry_policy=RetryPolicy(max_attempts=4, retry_on=is_transient),
)

is_transient says yes to timeouts, 5xx and 429, and no to everything else. Retrying a 401 four times is just failing slower. Four attempts with backoff push a low-single-digit flake rate down into noise; that vendor hasn't killed a run since mid-May. The node body, meanwhile, went back to being business logic.

One thing the policy can't see: the vendor's other failure mode is a 200 with an empty body. No exception, no retry, garbage flowing downstream looking perfectly truthy. The node now validates the payload and raises on emptiness, deliberately, so that failure enters the same machinery as the honest ones. Retry policies handle exceptions; turning bad data into an exception is still your job.

fallback edges and the dead-letter key

When retries exhaust, my nodes don't raise anymore. They write {"source": ..., "error": ..., "attempts": 4} into a failures key with an append reducer, and a conditional edge after each risky node reads the state and routes: clean result onward, error to a fallback node. Two fallbacks run in production: a secondary data vendor that's slower but rarely down, and a summarizer that drops to a cheaper model when the primary provider is having a day.

I put this in topology instead of a try/except with a second API call inside it, because legibility matters under stress.

A fallback edge shows up in the trace. A try/except inside a node body is invisible at 2am.

When a report looks thin, I can see that the run took the fallback path (in the trace, in the graph diagram, in the state itself) without reading a line of node source. Failures that exhaust even the fallback land in the dead-letter key, and the compile node renders them as explicit gaps: source, attempts, timestamp. An hour later, a small repair graph reads the dead letters, re-runs only those checks, and patches the stored report. Most gaps heal before any human opens the file.

seven of nine: ship or retry

Fan-out makes failure partial by construction. A nightly run fans to nine sources; seven return, two die post-retry, and the pipeline has to decide at 3am with nobody awake whether to ship or hold.

I ship. With rules.

The two dead branches are almost always the flakiest, least load-bearing sources, and holding a whole report hostage for them wrecks your latency for the data analysts rank last anyway. A seven-of-nine report with gaps rendered explicitly is honest in a way my swallowing-era reports never were. The reader knows exactly what's missing and when it failed. And the repair graph usually fills the holes before morning coffee.

The exception is classified, not vibed. Sources carry a core or supplementary tag in config; a dead registry or sanctions check routes the run to hold-and-alert instead of publish, because those two checks are the reason the report exists at all. That classification took one afternoon with the client, and it's the most durable twenty lines of config in the project.

what it cost

The graph roughly doubled. Nine working nodes picked up retry policies, fallback nodes and a dead-letter tail, and the whiteboard version stopped fitting on the whiteboard. For a three-node toy this would be pure overhead: a bare try/except and a crash is fine when a rerun costs nothing. But most production agent failures are exactly these boring integration failures, not model failures, and topology is the only place I've found where error handling stays visible instead of rotting inside node bodies. The shape itself (isolate, then degrade, then record, then repair) isn't LangGraph-specific at all; it's resilient architecture wearing graph clothes.

The pipeline died at 2am twice last week. Both times it healed itself by 3, and I found out from the morning summary instead of from the client.