qa-evaluation

QA Evaluation

Guidelines for refining test plans based on actual implementation and evaluating results.

Phase 2A: Refinement Process

1. Analyze Git Diff

git diff HEAD~1 --stat          # Overview of changed files
git diff HEAD~1 -- <file>       # Specific file changes
git log -1 --oneline            # Commit message context

2. Compare Intent vs Reality

• Original acceptance criteria: What was planned
• Git diff: What was actually implemented
• Identify:
  • Selectors that changed (testids, classes)
  • Features added/removed
  • UI structure differences

3. Refine Test Steps

Update test commands to match actual implementation:

• Correct element selectors
• Adjust wait times
• Update expected outcomes
• Add tests for new features
• Remove tests for dropped features

Refinement Output Format

{
  "actual_implementation": "Brief summary of what git diff shows was implemented",
  "refined_test_steps": [
    {
      "id": "QA1",
      "criteria_id": "AC1",
      "description": "Updated description",
      "commands": [
        "agent-browser open http://localhost:1420",
        "agent-browser wait --load",
        "agent-browser is visible [data-testid='actual-element']",
        "agent-browser screenshot \"$RALPHX_AGENT_SCREENSHOT_DIR/qa1.png\""
      ],
      "expected": "Updated expectation"
    }
  ]
}

Phase 2B: Test Execution

Execution Flow

1. Execute each test step in order
2. Record pass/fail for each command
3. Capture screenshot regardless of outcome
4. Continue testing even if step fails
5. Calculate overall status

Result Recording

For each step:

• status: "passed" | "failed" | "skipped"
• screenshot: Path to captured image
• actual: What actually happened (if failure)
• expected: What should have happened
• error: Error message (if any)

Test Results Output Format

{
  "qa_results": {
    "task_id": "task-123",
    "overall_status": "passed",
    "total_steps": 5,
    "passed_steps": 5,
    "failed_steps": 0,
    "steps": [
      {
        "step_id": "QA1",
        "status": "passed",
        "screenshot": "$RALPHX_AGENT_SCREENSHOT_DIR/qa1-result.png",
        "actual": null,
        "expected": null,
        "error": null
      }
    ]
  }
}