llm-testing-expert

LLM Testing and Evaluation

Description

Provide LLM test strategy design, test dataset construction, automated evaluation, and security red teaming recommendations to ensure models meet production requirements for functionality, performance, robustness, and safety.

When to Use

• User requests "design LLM testing strategy" or "how to evaluate model quality"
• User needs to build test datasets, design test cases, or perform regression testing
• User seeks prompt optimization, engineering, and version management recommendations
• User needs to evaluate model performance metrics (accuracy, hallucination rate, safety, latency, cost)
• User asks how to conduct A/B testing or comparative evaluation (multi-model/multi-version)
• User needs automated testing workflows or CI/CD integration
• User requests red team testing design (jailbreak, prompt injection, adversarial attacks)
• User asks about specialized testing strategies for RAG systems or Agent applications

When NOT to Use

• User only needs traditional software testing (unit tests, integration tests) without LLM-specific concerns
• User's problem is about model training or fine-tuning techniques, not testing/evaluation
• User needs data annotation or data cleaning, not test strategy design
• User only asks how to use an LLM framework or tool, not testing strategy
• User's problem is pure Prompt Engineering (how to write better prompts), not how to test prompt effectiveness

Input

{
  testTarget: {
    type: string                  // Test object type (base-model/fine-tuned-model/rag-system/agent/prompt-template)
    modelInfo?: {
      name: string                // Model name (e.g., gpt-4, claude-3.5-sonnet)
      version?: string            // Version
      deployment?: string         // Deployment method (API/self-hosted)
    }
    applicationContext?: string   // Application scenario (e.g., customer service, code generation, document summarization)
  }
  testObjectives: {
    functional?: boolean          // Functional correctness testing
    performance?: boolean         // Performance testing (latency, throughput, cost)
    robustness?: boolean          // Robustness testing (adversarial samples, edge cases)
    safety?: boolean              // Safety testing (jailbreak, injection, PII leakage)
    userExperience?: boolean      // User experience testing
  }
  constraints: {
    budget?: string               // Testing budget (API call cost/manual annotation cost)
    timeline?: string             // Testing timeline
    existingTestAssets?: string[] // Existing test assets (test sets, annotated data)
    complianceRequirements?: string // Compliance requirements (GDPR, industry regulations)
  }
  riskAreas?: string[]            // Known risk areas (e.g., hallucination, bias, privacy leakage)
  existingMetrics?: string        // Existing evaluation metrics and baselines
}