Streaming through chains without losing your mind

The ops copilot I run for a retail client answers questions about orders and inventory in eight to fourteen seconds end to end. Nobody wants to stare at a spinner for fourteen seconds, and nobody has to. The same latency feels fine when words start arriving inside the first second. Streaming is the cheapest UX win in this field, which makes it genuinely annoying how easy it is to break by accident.

the ux math nobody writes down

People read at maybe four or five words a second. A model producing a 300-word answer emits faster than that. So when the first token lands quickly, the reader starts immediately and never catches up to the generation front, and total latency disappears behind human reading speed. Time to first token is the metric that matters in chat surfaces, and I've mostly stopped optimizing anything else there. The systems version of this argument runs the numbers properly.

Mostly. A long tool call that stalls the stream mid-sentence still reads as a hang, which is what the event stream further down is for.

the parser that ate my stream

LCEL propagates streaming through a chain as long as every step can transform a stream instead of demanding its whole input. My chain streamed beautifully in April. In early May I appended a small RunnableLambda to tidy whitespace and normalize markdown headings before the frontend saw them.

A plain RunnableLambda consumes its entire input before it runs. The chain still worked (same outputs, tests green) but the UI now sat silent for nine-ish seconds and then dumped the whole answer at once. The client's PM mentioned the assistant felt slower this week. I blamed the provider, refreshed status pages, and spent four days being confidently wrong before I finally measured time to first token: around 600 milliseconds before my change, the full response time after it.

Every step after the model either passes the stream through or quietly turns your product back into a batch job.

The repair took an hour: cleanup moved client-side, and the one transform that had to stay on the server became a generator that yields as it receives. The lesson cost four days. Streaming is a whole-chain property, and composition is exactly where LCEL bites.

I now assert time to first token in CI, because nothing else guards it.

astream_events, the firehose

For anything beyond raw tokens (retrieval progress, tool starts, the small theater that keeps a nine-second answer from feeling dead), astream_events is the right tool and an aggressively noisy one. It emits events for every component in the chain, so you filter hard:

async for ev in chain.astream_events(question):
    if ev["event"] == "on_chat_model_stream":
        send(ev["data"]["chunk"].content)

I tag the two or three runnables the UI actually narrates and ignore everything else. I've had to revisit those filters after upgrades twice now, which tells you how much I trust the event surface to hold still. Worth it anyway: a searching-the-catalog line appearing at second one changes how the entire wait feels.

structured output hates streaming

with_structured_output hands you a validated object when generation finishes, which is precisely when streaming stops being useful. For a while I streamed raw JSON and parsed it in the frontend. Ugly and brittle; abandoned within a week.

The compromise I've settled on splits the response. The model writes the prose answer first, streamed token by token, and a compact structured block with citations and follow-ups arrives at the end. If you truly need one envelope, partial-JSON parsing can render an answer field as it grows, but you inherit every half-parsed edge case in your UI code.

Stream the words, batch the facts.