How to evaluate AI agent memory

Evaluation boundary

What should an agent memory evaluation isolate?

An agent memory evaluation should isolate memory-module quality, agent behavior, environment outcome, and operated-system behavior. Correct retrieval can still produce a bad answer. A correct decision can still meet host or worker failure. No layer stands in for another, so each case records the earliest divergence.

Failure attribution should follow the owned boundaries in the agent memory architecture guide. Record expected state before a run, then name the first component whose observable output differs.

Synthetic conformance fixture

Downloadable LANTERN-1 fixture

Download LANTERN-1 as JSON Media type: application/json. Suite: LANTERN-1-20260828. Released-byte SHA-256: ff2551ea54b0065ea778eaf990ddaacdddc213d926a15c5b6eed89add22d6afc.

Fixture duties in execution order

1. Reset the adapter and start the virtual clock.
2. Ingest scoped synthetic events and record formation operations.
3. Recall and answer current, valid-time, stale, and foreign evidence probes.
4. Decide before confirmation, record one scoped attestation, then decide again.
5. Execute with scope, actor, action, decision, expected state, and one retry key.
6. Read downstream state independently after the committed effect.
7. Report a receipt with that independent state, then verify receipt truth.
8. Delete one superseded record with its own idempotency key.
9. Erase one scope and scan every declared lifecycle surface.

Compact valid JSON excerpt

{
  "schema_version": "1.0",
  "suite_id": "LANTERN-1-20260828",
  "adapter_contract": {
    "execute": {
      "input": ["scope", "actor", "action", "decision_id", "expected_state", "idempotency_key"],
      "output": ["effect_id", "authorization_result", "state_check_result"]
    },
    "read_state": {
      "input": ["scope", "actor", "resource", "read_at"],
      "output": ["independent_state"]
    },
    "report_receipt": {
      "input": ["scope", "effect_id", "host_report", "independent_state", "idempotency_key"],
      "output": ["receipt_id", "receipt_truth"]
    },
    "delete_record": {
      "input": ["scope", "actor", "record_id", "expected_version", "idempotency_key"],
      "output": ["operation", "deleted_record_id", "retained_record_ids"]
    }
  }
}

Reproducibility record

Illustration Pin the fixture hash, adapter version, model ID, prompt, retrieval and context budgets, judge anchors, exclusions, seed, and virtual-clock policy. Retain raw outputs and operational failures with the metric vector. A changed model, index, policy, dependency, or host path creates a new run identity.

External research The peer-reviewed Datasheets for Datasets article motivates dataset documentation E17. Pineau's peer-reviewed JMLR article reports NeurIPS reproducibility practices E18. Neither source defines LANTERN fields or run identity.

Formation metrics

How should memory formation be scored?

Score memory formation by comparing expected and observed ADD, UPDATE, DELETE, and NOOP decisions, then inspect candidate content and binding. Operation macro F1 averages class-level scores across those decisions. It cannot show whether a retained fact has the correct source, target, time, privacy treatment, or downstream value.

NOOP matters because an indiscriminate writer should not gain credit for retaining every turn. Report class support and a confusion matrix beside macro F1 so rare DELETE cases do not disappear inside an aggregate.

Retrieval and answer metrics

How should retrieval and answers be scored separately?

Score retrieval against required, stale, and forbidden evidence before scoring its answer. A correct answer can hide poor retrieval, while correct retrieval can yield unsupported claims. Grounding links claims to cited evidence, but it does not prove that evidence is true, current, or authorized for action.

External research A peer-reviewed ACM Transactions on Information Systems article grounds normalized discounted cumulative gain E01. Peer-reviewed NAACL work on KILT couples answers with provenance in its benchmark E02.

External research Rashkin and colleagues' peer-reviewed Computational Linguistics article defines AIS as Attributable to Identified Sources E03. AIS evaluates whether output has identified support. It does not establish source truth. Peer-reviewed EMNLP work on FActScore demonstrates atomic decomposition in its published setting E04.

Lifecycle and security

How should updates, isolation, and erasure be tested?

Test updates, isolation, and erasure at named checkpoints after writes, corrections, conflicts, retries, deletions, restarts, and restores. State checkpoint accuracy compares observed with expected memory after an event. An end-state pass can miss earlier leakage, stale answers, duplicate versions, or over-erasure.

Add corrections, same-time conflicts, expiry, suppression, cross-scope reads and writes, reindexing, cache refill, worker restart, and backup restore. A physical-erasure claim reaches no farther than the declared surfaces and deadline that were scanned. The agent memory security guide covers the related threat model.

Action and operations

How should action and retry failures be measured?

Measure action and retry failures at decision, host, environment, retry, and receipt boundaries. When a system exposes act, ask, and abstain, score their confusion matrix separately from tool choice and arguments. Those outcomes do not establish authorization or execution. The host must check permission, live state, and business rules.

Retry the same key with equal and conflicting payloads. Receipt truth means agreement between a host-reported receipt and an independent state read within the declared check. Report operational error as a distinct non-pass state so a timeout cannot be counted as either safe behavior or a behavioral regression.

Benchmark coverage

Which public benchmark fits each memory question?

Choose a public benchmark by its measured ability, then add a private fixture shaped like the deployed workload. Results remain bounded by dataset version, model, prompts, retrieval budget, judge, and task environment. No listed benchmark covers formation, retrieval, lifecycle, isolation, action, operated recovery, and erasure together.

Pin the MemoryAgentBench artifact commit beside the paper terminology when reproducing that work. External research E08

Release gates

How should memory tests block a release?

Memory tests should block releases through explicit gates tied to risk, ownership, and cadence. Deterministic safety assertions fail independently of baseline comparison. An unchanged baseline cannot excuse foreign-scope access, unauthorized action, stale answers, duplicate effects, false receipts, or erasure residue. Operational failure remains a separate non-pass state.

Store the earliest failing boundary, reproduction inputs, affected scope, and owner with each result. NIST AI Risk Management Framework 1.0 supports documented measurement practice, but it defines no agent-memory score or release threshold. External research E16

Bounded product mapping

Current ContextDB mapping

Current ContextDB behavior

At repository snapshot d59c1b9, the Memory CI command-line package is Available and hosted Memory CI is Hosted Alpha. Memory CI runs bounded deterministic recall and action cases against a pinned baseline. It is a customer regression tool, not a public scientific benchmark.

This neutral guide covers formation scoring, retrieval, grounding, lifecycle checkpoints, public benchmark selection, faults, receipts, latency, cost, and erasure assertions. The product Memory CI page owns its current workflow and behavior. Current hosted operation does not establish scale, availability, formation scoring, physical erasure, downstream side-effect truth, or cross-vendor comparability. LANTERN-1 is not a ContextDB contract.

Evaluation FAQ

Questions about agent memory evaluation