The field guide
What agent memory is, why context windows are not it, how memories should form, evolve, and retrieve, and where trust and privacy fit. Drawn from an analysis of over 200 research papers and from operating memory in real customer-facing agents. Written for the engineer who has to ship one.
Definitions
An LLM is stateless. Every inference call starts blank, bounded by a finite context window. Agent memory is the system outside the model that carries knowledge across calls, sessions, channels, and time: who this user is, what was said last week, what worked last time, and what is still unresolved.
A long context window is a bigger blank slate, not memory. It forgets at the end of the call, costs tokens on every request, and degrades in the middle: models under-attend to information buried in the center of long prompts.
RAG retrieves relevant documents for a query. Memory tracks facts about specific users and outcomes of past actions, with provenance and a lifecycle. The two overlap in mechanics and differ in job. See memory vs RAG.
A user profile table stores state. It does not decide what is worth remembering, notice when a stored fact goes stale, or distinguish a confirmed instruction from an overheard maybe. That intelligence is the actual work.
The status quo
The default architecture in 2026: a vector database for embeddings, Redis for session state, Postgres for user profiles, object storage for transcripts, and glue code to stitch them together. Each component is fine. The system fails at exactly the capabilities agents need most, because no component owns the memory lifecycle.
| Patchwork piece | What breaks |
|---|---|
| Vector store | Semantic-only. No temporal, causal, or entity awareness. |
| Redis session state | Ephemeral. Last week's call is gone. |
| Postgres profiles | Static rows. No lifecycle, no idea which fact is stale. |
| Transcript archive | Write-only. Not queryable by meaning or time. |
| Glue code | Brittle, duplicated across teams, owned by no one. |
| Missing entirely | Trust: nothing decides what the agent may act on. |
The taxonomy
The research literature has converged on a taxonomy of memory functions (Hu et al., 2025, surveying 200+ papers). Most deployed systems implement one and call it done.
Declarative knowledge: who the user is, what they prefer, what they confirmed. This is the most mature function commercially, and the one with the sharpest trust problem, because factual memory is what agents act on.
"prefers morning appointments" · user_stated · 0.95What worked and what failed: resolution patterns, escalation outcomes, lessons from bad calls. The most underserved function in the literature, and the one that compounds: a library of 100 learned workflows generalizes in a way 10 never can.
"offered afternoon only; caller frustrated; ask preference first"The active context of the current task: what is on the table right now, compressed and paged to respect the token budget. Pioneered by MemGPT-style context paging; still where most token waste hides.
open issue: compressor noise · awaiting parts ETADynamics 1 of 3 · Formation
The most common failure mode in memory-augmented agents is not missing memories. It is too many low-quality ones. Store every turn and the memory fills with "thanks for calling" and "let me check that." Every retrieval then wades through noise, and the answer the agent needed ranks below its retrieval window.
In pilot deployments reported in the ContextDB research paper, agents with unfiltered turn-level memory performed worse than agents with no memory at all after roughly 200 conversations, because retrieval precision collapsed under noise. The counterintuitive fix, established by SeCom and confirmed independently since: segment conversations into topically coherent units, extract the atomic facts, and compress before storing. Compression is denoising, not loss. Removing filler before embedding produced 12 to 18% better retrieval precision in published evaluations, and cuts the tokens injected into every future prompt by 80 to 90%.
Rule of thumb: a support agent handling 50 conversations a day produces roughly 500 turn-level items daily and 15,000 a month, most of them noise. Segment-level formation with compression yields around 2,000 high-signal items from the same month, and the facts that matter rank at the top instead of position ten.
# raw turn "I called last week about my AC. The tech came Tuesday but the compressor is still making noise." # segment → extract → compress factual: "AC compressor issue, unit 4B" experiential: "tech visit did not resolve" working: "open issue, follow-up needed" # stored: 3 dense items, not 14 raw turns
Dynamics 2 of 3 · Retrieval
"What is Alex's email?" is an entity lookup. "What happened after the repair?" is temporal traversal. "Why did the escalation fail?" is causal chain following. "Find calls like this one" is semantic similarity. Embedding similarity alone, the default in most stacks, fails silently on the first three: it returns related-sounding but temporally or causally wrong memories.
This matters more than benchmark deltas suggest. In the AI phone-agent deployments analyzed in the paper, temporal queries were the single largest category of memory retrievals that changed agent behavior, roughly 45% of impactful cases: "when was the last service visit?", "how many times has this recurred?", "what was promised on the last call?" These are precisely the queries flat vector similarity handles worst.
The literature's answer is converging: graph-structured memory. Semantic, temporal, causal, and entity relations as separate views over the same memories, with retrieval weights chosen per query type. Graph-based memory consistently beats flat vector stores on multi-hop and temporal reasoning across HippoRAG, GraphRAG, Zep, and successor systems.
# "what happened after the AC repair?" query_type = temporal weights = { semantic: 0.2, temporal: 0.5, causal: 0.3, entity: 0.0 } # "what is Alex's email address?" query_type = entity weights = { semantic: 0.3, temporal: 0.0, causal: 0.0, entity: 0.6 }
Dynamics 3 of 3 · Evolution
Facts go stale: the customer moves, the plan changes, the policy updates. Restatements pile up as duplicates. New information contradicts old. A store that only appends will eventually answer "where does this customer live?" with whichever address happens to embed closer to the query.
Same entity, same attribute, new value is an update, not a second fact. Learned memory managers that choose between add, update, delete, and no-op beat heuristics decisively in published work, with surprisingly little training data.
Track when a thing happened and when the system learned it, separately. "I called last week about this" has an event time of last week and an ingestion time of now. Conflate them and temporal questions become unanswerable.
The paper reports a curated store of ~300 managed memories outperforming a raw store of 3,000 by 15 to 20% F1 on LoCoMo-style evaluation. The mechanism is retrieval precision: fewer, cleaner memories mean the top results are the right ones.
The missing dynamic · Trust
Formation, retrieval, and evolution decide what the agent knows. None of them decide what the agent may do. The moment a remembered sentence authorizes a booking, refund, or account change, memory stops being a context problem and becomes a permission problem. Most memory systems store an overheard "maybe Friday" and a confirmed "Thursday, yes" with equal weight, and the tool call cannot tell them apart.
This is the layer ContextDB Cloud ships today: every memory carries a source and confidence, consequential recall returns only evidence that passes policy, and the outcome is an explicit act, ask, or abstain with the evidence recorded. Tentative facts wait in a confirmation queue until a human says yes. The decision log is the audit trail.
# conversational recall: everything relevant recall("schedule preferences") → "maybe Friday" (0.4), "Thursday ✓" (0.95) # pre-action recall: trusted evidence only recall_for_action("book the visit") → "Thursday ✓" (confirmed, user_stated) outcome = act evidence recorded # and when only the maybe exists: outcome = ask "which day should I book?"
Privacy
Published black-box attacks (MEXTRA, 2025) extract private information straight out of agent memory stores. Yet across 200+ surveyed papers, fewer than five treat privacy as a first-class concern. If PII reaches the store, it persists, is retrievable by anything with access, and outlives the conversation that produced it.
Application-level filtering fails because developers forget, edge cases slip through, and PII can be inferred from combinations of non-PII facts. Detection belongs in the write path of the memory layer, before anything embeds or persists.
Different memory types deserve different lifetimes, and "delete everything about this user" must actually delete everything: the memories, their graph links, and derived artifacts.
Every create, read, update, and delete logged with who, when, and why, append-only. When an agent acts on a memory, the decision and its evidence should be reviewable without replaying a prompt.
The checklist
From the paper's synthesis of the literature and production pilots. Score your current stack against these before adding a single feature.
| Prerequisite | The test |
|---|---|
| 1. Precise formation | Does storage filter and compress, or does every turn go in? |
| 2. Query-adaptive retrieval | Can it answer "what happened after X?" or only "what sounds like X?" |
| 3. Active evolution | When a customer states a new address, does the old one die? |
| 4. Token-budgeted working memory | Does context injection page and compress, or paste everything? |
| 5. Memory-layer privacy | Is PII caught in the write path, or trusted to app code? |
| 6. Memory boundaries | Do multiple agents get role-scoped views, or one shared soup? |
| 7. Live-call latency | Is recall fast enough for a phone call, precomputed where possible? |
Field notes
Directional evidence from pilot deployments across customer communication products, as reported in the research paper. Not controlled experiments; useful priors.
Recognizing that today's email is yesterday's caller reduced repeated information requests by roughly 35% and cut average handle time by about 20% in shared-inbox support deployments. The entity graph pays for itself here first.
For phone agents, resolving the caller and compressing their top memories into a short snapshot during the ring interval cut first-response latency from 2.1s to 0.4s. The agent opens the call already knowing the history.
Factual memory helped linearly. Resolution-pattern memory compounded: agents remembering what worked resolved similar issues about 40% faster than agents with profiles alone, and began resolving novel issues by analogy past ~500 stored patterns.
Questions
No. Long contexts forget at the end of the session, cost tokens on every call, and under-attend to the middle of the prompt. Memory with good formation and retrieval matched full-context accuracy within a few points in published evaluation while using a small fraction of the tokens, which also unlocks smaller, faster models for the same task.
RAG retrieves relevant knowledge for a query, usually from documents. Memory tracks user-specific facts and experience with provenance and a lifecycle, and, when actions are involved, decides what may be relied on. Keep RAG for knowledge. Add memory for continuity and trust. See the full memory vs RAG breakdown.
Extracted and compressed facts. Raw transcripts belong in your transcript store. Turn-level raw storage measurably degrades retrieval as volume grows; segment-level extraction with compression improves it. Keep the pointer to the transcript as evidence, not the transcript as the memory.
The moment a recalled statement can authorize a consequential action: a booking, refund, credit, account change, or workflow mutation. Until then, relevance-ranked recall is enough. After that, you need source, confidence, confirmation status, and an explicit act, ask, or abstain outcome. That gate is what ContextDB ships.
Track both statements with their event and ingestion times, treat the newer confirmed statement as current, and keep the history. For consequential actions, an unresolved contradiction should surface as ask, not silently pick a winner.
Public benchmarks (LoCoMo, LongMemEval) test conversational recall, multi-hop reasoning, and temporal ordering. They do not test your domain: issue recurrence tracking, escalation quality, booking accuracy. Build a small eval set from your own decision log, and measure retrieval precision, not just recall. More memory is not better memory.
Store one sourced fact, gate one consequential action, read the decision.