AgentEval memory-benchmarks.instructions.md

AgentEval Memory & Benchmarks — Agent Instructions

Module Overview

The AgentEval.Memory module provides comprehensive memory evaluation for AI agents:

• Native benchmarks (12 scenario types across 5 presets)
• External benchmarks (LongMemEval — ICLR 2025, 500 questions, 6 types)
• Baseline persistence with JSON file storage
• Interactive HTML reports with pentagon/radar charts, timeline, and comparison
• History injection modes (Auto, Structured, TextBlob)

Architecture

src/AgentEval.Memory/
├── Abstractions/       Interfaces for memory operations
├── Assertions/         Memory-specific fluent assertion API
├── Data/               Embedded datasets, LongMemEval data files
├── DataLoading/        Scenario & corpus loaders, exporters
├── Engine/             Core: MemoryTestRunner, MemoryJudge
├── Evaluators/         MemoryBenchmarkRunner, ReachBack, Reducer, CrossSession evaluators
├── Extensions/         DI registration, helper methods
├── External/           External benchmarks (LongMemEval), interfaces
├── Metrics/            Memory-specific evaluation metrics
├── Models/             Core data: Benchmark, Result, Baseline, Config
├── Report/             HTML report template, pentagon mapper
├── Reporting/          Baseline store, comparer, output formatting
├── Scenarios/          Scenario providers: Memory, Chatty, Temporal, CrossSession
└── Temporal/           Temporal memory runner & scenarios

CRITICAL: Never Interrupt Running Benchmarks

NEVER kill, cancel, or interrupt a benchmark that is in progress. LLM benchmarks make hundreds of API calls and can take 10-100+ minutes to complete. Killing a benchmark wastes all progress, API costs, and time.

• If you need to add console output, logging, or cosmetic changes — wait for the benchmark to finish first, then make changes and re-run.
• If a benchmark appears to produce no output, it may still be running. Check the terminal for signs of life (CPU usage, network activity). The runner logs progress via Console.WriteLine after each question.
• If you want to add progress reporting to a runner that lacks it, wait for the current run to complete.
• The only valid reason to interrupt is an explicit user request or an unrecoverable crash.