omnimemeval-eval

OmniMemEval evaluation

Read this before the first run, after forgetting the workflow, or when resuming a run. The stable source of truth is evals/omnimemeval/benchmark.config.json; do not reconstruct its parameters on the command line.

Run

For an embedding-backed run, start the single local service in a separate terminal with the already prepared interpreter, then verify that /health reports "device":"cuda":

.benchmarks/bge-venv/Scripts/python.exe evals/omnimemeval/bge_server.py --device cuda

Do not use uv run --with for this service on Windows: it can resolve a second, CPU-only PyTorch environment. Omit --device only when automatic GPU-or-CPU selection is intentional; use --device cpu for an explicit CPU run.

# Inspect without model or embedding work.
npm run benchmark:omni -- longmemeval --dry-run

# Run one complete official suite.
npm run benchmark:omni -- longmemeval

Supported suite names are longmemeval, locomo, beam, personamem-v2, and halumem. The runner generates a unique version, loads the configured env file, establishes the NMG/UTF-8/venv environment, and delegates to the pinned official script.

Configure once

Edit the checked-in config when the experiment policy changes:

{
  "envFile": ".env.nmg-opencode",
  "commonArgs": ["--workers", "16", "--llm-workers", "16", "--top-k", "20"],
  "suites": {
    "longmemeval": [],
    "beam": ["--scale", "100k"]
  }
}

commonArgs apply to every suite. suites.<name> forwards only that suite's official options. Keep provider keys, model names, embedding endpoints, and QPP configuration in the env file. Do not put runner-owned --lib, --env, --version, or --replay flags in the config.

For a one-off canary, copy the config and select it explicitly:

npm run benchmark:omni -- beam --config evals/omnimemeval/canary.config.json

Do not add a CLI flag merely to avoid editing or copying the config.

Resume