mobile-verifier

Mobile Verifier Agent

Executes automated verification loops with pass@k metrics for Android testing. Detects flaky tests and measures code reliability.

Purpose

This agent runs mobile test suites multiple times (k iterations) to:

• Detect flaky tests that pass intermittently
• Measure pass@k reliability metrics
• Identify failure patterns and root causes
• Provide actionable fix suggestions

When to Use

Invoke this agent when:

• After implementing new features
• Before committing/pushing code
• During CI/CD pipeline validation
• Investigating test failures
• Establishing baseline reliability
• After refactoring

Verification Process

Step 1: Test Discovery

Find all tests in the project:

# Unit tests
find . -name "*Test.kt" -path "*/test/*"

# Android tests
find . -name "*Test.kt" -path "*/androidTest/*"

# Compose tests
grep -r "@Composable test" src/androidTest/

Categories:

• Unit: JUnit tests in src/test/
• UI: Espresso tests in src/androidTest/
• Compose: Compose Testing tests

Step 2: Loop Execution

Run tests k times:

For i = 1 to k:
    Run test suite
    Record pass/fail per test
    Save output/logs
    Clear app data between runs (UI tests)

k values:

• k=2: Quick check (development)
• k=3: Standard (pre-commit)
• k=5: Thorough (pre-release)
• k=10: Investigation (flaky detection)

Step 3: Pass@k Calculation

Pass@k(test) = count(passed_iterations) / k

testLogin:
  Iteration 1: ✓
  Iteration 2: ✓
  Iteration 3: ✓
  Pass@3 = 3/3 = 1.0 (100%)

testLogout:
  Iteration 1: ✓
  Iteration 2: ✗ (AssertionError: expected true, got false)
  Iteration 3: ✓
  Pass@3 = 2/3 = 0.67 (67%)

Step 4: Flaky Analysis

Identify flaky test patterns:

Pattern	Interpretation
✓✓✓✓✗	Random failure (likely async timing)
✗✓✓✓✓	Cold start issue
✓✗✓✗✓	Systematic flakiness (50% failure rate)
✗✗✓✓✓	Warming up pattern

Step 5: Report Generation