Conversation memory: buffers, summaries, and what I actually use

The freight-tracking support bot I built for a logistics client went through three memory architectures between December and April. Each one failed in public, because public is the only place conversation memory ever really fails.

the buffer era: pay per turn, forever

Version one did what every tutorial does: append the entire history to every request. Nothing gets forgotten, which sounds like the goal until you watch it run. By turn 40 of a gnarly customs dispute the prompt weighed around 19,000 tokens, latency had roughly doubled since the start of the session, and answer quality was sliding in that diffuse way long contexts slide. The model was attending to everything and committing to nothing. I've written about why big windows don't save you; this bot was the live demonstration. The meter also runs per turn, so the long angry conversations that matter most are the ones that cost most.

windows: amnesia with a cliff

Version two kept the last 12 turns and dropped the rest. Cheap and fast, and it produced the worst bug report of my career: the client's support lead wrote, in more professional wording, that the bot was gaslighting customers.

She wasn't wrong.

A customer had been promised a fee waiver around turn 9. By turn 24 the promise had slid out of the window and the bot flatly denied it existed. From the customer's side, that's bad faith with a friendly tone. Window memory has a cliff, and the conversations long enough to matter are exactly the ones that fall off it.

the demo where it forgot her name

Version three was summary memory: a model compresses older turns into a rolling paragraph that rides along in the prompt. I argued for this one in the client's planning call, out loud, with slides. I want that on record, because a week later it humiliated me in front of the people I'd convinced.

Their head of support ran a realistic case in the demo: introduced herself by name, gave an order number, walked through a delayed shipment. Twenty-odd turns later, the rolling summary had compressed her introduction down to a customer reporting a late delivery. The bot asked for her name again. Three turns after that, it asked for the order number it had already been given.

You can watch trust leave a room, and it moves quickly.

Summarization is lossy in precisely the wrong direction: it keeps the gist and drops the name, the order number, the promise with a date on it. It also adds a model call every turn, so I was paying extra latency for the privilege of losing information.

what I run now: state, not memory

Memory is the part of your data model you hoped a framework would design for you.

The current bot has no memory class at all. It has state: a typed dict holding the customer's name, their order ids, and every commitment the bot has made, each one dated.

class SessionState(TypedDict):
    customer_name: str | None
    order_ids: list[str]
    promises: list[str]  # dated

A small extraction step updates that state after every turn. The prompt renders the state block plus the last eight turns verbatim, so recent detail stays exact and older detail survives as structured fields instead of prose soup. A LangGraph checkpointer persists all of it (sqlite on my Mac Studio, postgres in production), so a restarted pod resumes mid-conversation instead of greeting the customer like a stranger. Anything that should outlive the session graduates into long-term memory, which is a separate discipline with separate failure modes.

The honest cost: the extractor is one more cheap call per turn, and it misses roughly one oddly-phrased commitment in twenty sessions. We log its diffs and review the misses weekly, which is tedious and fine. An auditable miss beats a silent summary, and the broader architecture argument makes that case at essay length.

Their head of support signed off in May. Nobody has been asked their name twice since.