What an agent should remember, and what it should be made to forget

The word "memory" makes engineers reach for a vector database. Someone decides their agent should "remember things," and within a day there's an embedding store, a retrieval pipeline, and a diagram with a box labeled Long-Term Memory. Most of the time that whole apparatus is solving a problem the project doesn't have yet, and it's a distraction from the part that's actually hard.

Let me untangle what "memory" even means here, because three different things wear the name and they have nothing to do with each other architecturally.

The first is short-term memory, and it's just the conversation sitting in the context window. When people say the agent "remembers what I said earlier in this chat," that's not a memory system, that's the window holding the transcript. Managing it well, keeping it lean, not letting it fill with stale junk, is real work, but it isn't a database.

The second is the working scratchpad: notes the agent keeps for the current task, often literally a file it writes to as it goes. It survives across the steps of one job and lets the agent track what it's tried and learned without holding all of it in the window at once. This is cheap and underused. A plain file is usually the whole implementation.

The third is the one people mean by "memory," durable facts that persist across sessions. Conventions, decisions, a user's standing preferences, the lessons learned last week. This is the layer worth thinking carefully about, and it's also the one that gets over-built.

You probably don't need a vector store for it

Here's the contrarian bit. Most agents with durable memory don't need a vector database to hold it. A markdown file, a handful of curated notes, a CLAUDE.md at the repo root, that's a perfectly good long-term memory for an enormous range of agents, and it has the lovely property that you can read it, edit it, and reason about it directly. You reach for a vector store over your memories only when you have so many of them that retrieving the relevant ones becomes its own search problem. Until then, the store is infrastructure you're maintaining for a scale you haven't reached.

The hard part is curation, not storage

Storing memories is trivial. Deciding what's worth storing is the entire job, and it's where memory systems live or die.

What's worth remembering is narrow: the corrections, the confirmed approaches, the durable preferences, the hard-won fact that took the agent three tries to discover. One lesson per note, with a line about why it mattered, so future retrieval has something to latch onto. What's not worth remembering is most of everything else, the transient details of one task, the obvious, and especially anything already recorded in the code or the git history. Saving those is memory bloat, and bloat dilutes the signal exactly the way a junk-filled context window does. A memory store full of trivia is harder to retrieve well from, not more capable.

Then there's the failure mode nobody plans for: a wrong or stale memory is worse than no memory at all. If the agent durably remembered something that used to be true and now isn't, it will act on that false fact confidently, and the error propagates into every future session that retrieves it. This is the staleness problem that haunts documentation, now wired directly into the agent's behavior. Durable memory therefore needs maintenance as a first-class activity: update what changed, correct what was wrong, and delete what stopped being true. A memory you only ever append to is a memory that slowly fills with confidently-wrong facts.

How it actually plugs in

When you do have durable memory, the agent has to retrieve from it to use it, which means you're doing RAG over your own memory, inheriting all the same retrieval failure modes. Recency complicates it further: a correction from yesterday should usually outweigh a note from six months ago, so pure similarity isn't enough, you want recency in the ranking. And whatever you retrieve gets placed into the context window, which means memory is subject to the same context discipline as everything else, relevant, well-placed, and not so voluminous it drowns the task.

The stack I'd actually build, in order: the conversation for short-term, a scratchpad file for the current task, and a small, curated, regularly-pruned long-term store, markdown or a simple table, that you only upgrade to a vector store when its size genuinely demands retrieval. Add the embedding machinery when a real problem forces it, not because "memory" sounded like it needed a database.

The reframe worth keeping is that agent memory is a curation discipline wearing a storage costume. The engineering question that looks important, where do we store the vectors, is the easy one. The questions that decide whether the memory helps or hurts, what's worth keeping, how do we keep it true, what do we throw away, have nothing to do with the database and everything to do with judgment. Teams that over-invest in the store and under-invest in the curation end up with a large, well-indexed pile of stale facts, and an agent that recalls them all with total confidence.