ContextDB Evals · Private Hosted Alpha

Catch memory regressions before your agent reaches production.

Save deterministic recall and action cases, group them into encrypted suites, pin a successful baseline, run every case through the real gateway, and inspect exactly what changed.

Open Memory Testbench Review the memory API

The direct answer: ContextDB Evals verifies memory behavior with deterministic assertions. It does not ask an LLM to grade its own output.

The failure it catches

A model, policy, prompt, or memory change can alter what the agent recalls.

A green application test can still miss the memory that disappeared, the stale evidence that returned, or the action outcome that changed. Evals records those objects directly.

Save cases query · partition · assertions

Create suite encrypted case membership

Run all bounded real-gateway execution

Pin baseline successful immutable run

Compare unchanged or regressed

Deterministic cases

Say what must appear, what must not, and whether the agent may act.

Each case has a stable user partition, query, recall or action mode, result cap, optional confidence floor, expected phrases, forbidden phrases, and optional act/ask/abstain outcome.

A case with no assertion is observed, never passed. Suite cases require deterministic assertions.

case: scheduling preference
user_id: customer-123
query: Which day should I offer?
mode: action

must return:
  - Thursday

must not return:
  - Friday

expected outcome:
  act

Run All

Every dispatched case receives a terminal record.

Immutable snapshots

Suite execution freezes each encrypted case before dispatch. Editing a saved case later does not rewrite the history of a completed run.

Bounded execution

The current runner caps suites at 50 cases, four workers, 60 seconds overall, 15 seconds per case, and fixed result-size budgets.

Interrupted-run recovery

Console startup converts interrupted queued or running cases into explicit worker-loss errors and terminalizes the parent. It does not silently mark the suite passed.

Baseline comparison

A baseline is selected, never guessed.

Pin an explicit successful run. The next run compares status, evidence IDs, and action outcome for each case. The closed comparison states are no_baseline, unchanged, and regressed.

suite: support-actions
status: failed
regression_status: regressed

case: case_7f…
status: failed
evidence_ids:
  added:   [mem_new…]
  removed: [mem_old…]
action_outcome:
  baseline: act
  current:  ask

Private by construction

Exports report behavior without copying customer conversations.

Encrypted at rest

Suite names, case snapshots, comparisons, and summaries use project/object-bound authenticated encryption in PostgreSQL.

project + object AAD → encrypted payload

Content-free CI exports

JSON and JUnit include IDs, hashes, statuses, safe failure codes, counts, attempts, latency, and outcomes. They omit suite names, case names, queries, assertions, raw failures, and memory content.

behavior metadata · no transcript dump

Verified behavior

What has actually been proved.

Real PostgreSQL tests cover encrypted suite persistence, immutable snapshots, startup reconciliation, baseline semantics, and project isolation.
A 10-case action suite test proves eval traffic stays out of ordinary Decisions, production counts, webhooks, and billable action usage.
A bounded production proof passed baseline, unchanged rerun, deliberate regression, and privacy scans of JSON and JUnit exports.
Cleanup returned zero suite, suite-run, case-run, credential, secret, and orphan-key state.

Evals execution is still request-bound. There is no durable suite queue, scheduling, cancellation, public CLI, or GitHub Action. Startup recovery fails an interrupted run instead of resuming it.

Use cases

Test the changes that make memory drift.

Embedding migration

Compare the evidence returned before and after changing embedding models or dimensions.

Trust-policy change

Catch an action that moved from act to ask, or from ask to abstain.

Source backfill

Verify that new database context improves recall without reviving forbidden stale evidence.

Agent release

Run the same memory cases before shipping a new prompt, tool set, or workflow version.

Questions

Evals questions, answered directly.

Does Evals use an LLM judge?

No. The current MVP uses deterministic expected text, forbidden text, and action-outcome assertions.

Does an action eval execute the business action?

No. It evaluates memory policy through the real gateway but does not call your booking, payment, refund, or account tool.

Can I run suites from CI?

You can export JSON or JUnit from completed console runs. A public CLI, GitHub Action, and unattended scheduling are not shipped.

Does a crashed suite resume?

No. Startup reconciliation terminalizes interrupted work with explicit errors. Durable queued execution and resume are not part of this private alpha.

Break one memory on purpose before a release does it for you.

Create a case, pin a baseline, remove the evidence, and watch the suite report the regression.

Open ContextDB Evals Create memory asynchronously