runtime-reflexion

Runtime Reflexion (Error → Diagnose → Retry)

Context

Static skills freeze knowledge. Reflexion (Shinn et al. NeurIPS 2023) + Code Sandbox makes ML agents self-correct: 80%+ of runtime errors auto-fixable when errors are precisely classified. agy verified this as a P0 must-have for ML agents (MLE-bench Kaggle medal correlation).

The core insight: errors aren't random — they fall into ~12 categories, each with known fixes.

Guidance

Single Execution with Diagnosis

from framework.src.reflexion import CodeSandbox, ErrorAnalyzer, execute_with_analysis

sandbox = CodeSandbox(timeout_sec=30)
result, diagnosis = execute_with_analysis(generated_code, sandbox)

if not result.success:
    print(f"Error: {diagnosis.category.value}")
    print(f"Suggested fix: {diagnosis.suggested_fix}")
    print(f"Related skills: {diagnosis.related_skills}")
    # → "category=memory, fix=reduce batch_size, skills=[gpu-readiness-assessment]"

Self-Correction Loop

from framework.src.reflexion import ReflexionLoop, CodeSandbox

def my_llm_fix(prompt: str) -> str:
    """Call Claude/your LLM to fix code given the reflexion prompt"""
    return llm_client.messages.create(
        model="claude-sonnet-4-6",
        messages=[{"role": "user", "content": prompt}],
    ).content[0].text

loop = ReflexionLoop(
    sandbox=CodeSandbox(timeout_sec=60),
    llm_fix=my_llm_fix,
)
result = loop.run(initial_code, max_attempts=5)

if result.success:
    deploy(result.final_code)
else:
    log_to_vault(result.final_diagnosis)  # remember for next time

Error Categories (12)

Category	Trigger	Fix
syntax	SyntaxError	Check indentation
import	ModuleNotFoundError	pip install
shape	shape mismatch	reshape/transpose
memory	CUDA OOM	reduce batch_size
overfitting	val_loss diverging	dropout/early stop
data_leakage	"leakage" mentioned	check temporal features
timeout	timed out	reduce data/optimize
...	...	...