genotex-benchmark-guide

GenoTEX Benchmark Guide

Overview

GenoTEX is a benchmark for evaluating LLM-based agents on gene expression data analysis tasks. It provides curated datasets from GEO (Gene Expression Omnibus) with ground-truth analysis pipelines, testing agents on data preprocessing, differential expression, enrichment analysis, and biological interpretation. Published at MLCB 2025 as an oral presentation.

Benchmark Structure

GenoTEX Benchmark
├── Data Collection
│   └── Curated GEO datasets with ground truth
├── Task Categories
│   ├── Data preprocessing (QC, normalization)
│   ├── Differential expression analysis
│   ├── Gene set enrichment analysis
│   ├── Clustering and classification
│   └── Biological interpretation
├── Evaluation
│   ├── Code correctness (executes without error)
│   ├── Statistical validity (appropriate tests)
│   ├── Result accuracy (vs ground truth)
│   └── Interpretation quality (biological insight)
└── Baselines
    ├── GPT-4 agent
    ├── Claude agent
    └── Domain-specific fine-tuned models

Usage

from genotex import GenoTEXBenchmark

bench = GenoTEXBenchmark()

# List available tasks
tasks = bench.list_tasks()
for task in tasks[:5]:
    print(f"Task: {task.id}")
    print(f"  Dataset: {task.geo_accession}")
    print(f"  Category: {task.category}")
    print(f"  Difficulty: {task.difficulty}")

# Get a specific task
task = bench.get_task("GSE12345_DEG")
print(f"Description: {task.description}")
print(f"Input files: {task.input_files}")
print(f"Expected output: {task.expected_output_type}")

Running Evaluations

# Evaluate an agent on GenoTEX
from genotex import evaluate_agent

results = evaluate_agent(
    agent_fn=my_agent_function,
    tasks="all",            # or specific task IDs
    timeout_per_task=300,   # seconds
)

print(f"Tasks completed: {results.completed}/{results.total}")
print(f"Code correctness: {results.code_correct_rate:.1%}")
print(f"Statistical validity: {results.stats_valid_rate:.1%}")
print(f"Result accuracy: {results.accuracy:.3f}")