neuron-evaluation

Neuron AI Evaluation

This skill helps you create and run evaluations for AI systems in Neuron AI. The evaluation system provides dataset-driven testing with flexible assertions, comprehensive result reporting, and extensible output drivers.

Core Concepts

The Evaluation System

Evaluations test AI systems using three main components:

1. Evaluators - Test classes that define what to run and how to validate
2. Datasets - Test data sources (arrays, JSON files)
3. Assertions - Validation rules for checking outputs

Dataset Items → Evaluator::run() → Output → Evaluator::evaluate() → Assertions → Results

Evaluation Flow

For each dataset item:

1. setUp() - Initialize resources (once per evaluator)
2. run(datasetItem) - Execute your AI logic
3. evaluate(output, datasetItem) - Assert against expected results
4. Repeat for next item

Note: Each evaluation starts with a fresh assertion executor - no manual reset needed.

Creating Custom Evaluators

Basic Evaluator

use NeuronAI\Evaluation\BaseEvaluator;
use NeuronAI\Evaluation\Contracts\DatasetInterface;
use NeuronAI\Evaluation\Assertions\StringContains;
use NeuronAI\Evaluation\Dataset\ArrayDataset;
use NeuronAI\Agent;
use NeuronAI\UniqueIdGenerator;

class ContainsEvaluator extends BaseEvaluator
{
    public function getDataset(): DatasetInterface
    {
        return new ArrayDataset([
            [
                'text' => 'I love this product!',
                'content' => 'product',
            ],
            [
                'text' => 'This is terrible.',
                'content' => 'positive',
            ],
        ]);
    }

    public function run(array $datasetItem): mixed
    {
        // A fresh thread per item, so no item sees another item's conversation
        $response = MyAgent::make()
            ->setThreadId(UniqueIdGenerator::generateId('eval_'))
            ->chat(new UserMessage($datasetItem['text']))
            ->getMessage();

        return $response->getContent();
    }

    public function evaluate(mixed $output, array $datasetItem): void
    {
        $this->assert(
            new StringContains($datasetItem['content']),
            $output
        );
    }
}