evidence-standards

Evidence Standards for All Testing and Verification

Core Principle

Evidence must prove what you claim. Mock data cannot prove production behavior.

Evidence Class System

Evidence requirements vary by project type. Choose the class that matches your project:

Class	Examples	Required Evidence
Terminal/Backend	APIs, services, CLIs, scripts	run.json, metadata.json, evidence.md (git provenance + test output sufficient)
Integration	DB-backed services, external API clients	Terminal class + request_responses.jsonl + server runtime provenance
LLM/API	LLM integrations, model fine-tuning, API pipelines	Integration class + llm_request_responses.jsonl
UI/Browser	Web apps, rendered interfaces, user flows	Terminal class + UI/browser video evidence

For mixed-class projects, apply all applicable requirements. If your project has no browser UI, skip the browser video requirements. If your project has no LLM layer, skip the LLM request/response capture.

Evidence is NOT one-size-fits-all. A backend bug fix with python -m pytest output and git SHA is valid evidence. Do not generate video evidence for terminal-only work unless the work specifically requires it.

Minimum Viable Evidence Checklist

Every test MUST capture these at minimum (copy-paste into test setup):

def capture_provenance():
    """REQUIRED: Capture all evidence standards."""
    provenance = {}

    # === GIT PROVENANCE (MANDATORY) ===
    # For GitHub/GitLab with remotes: fetch first to ensure refs are current
    subprocess.run(["git", "fetch", "origin", "main"], timeout=10, capture_output=True)
    provenance["git_head"] = subprocess.check_output(
        ["git", "rev-parse", "HEAD"], text=True).strip()
    provenance["git_branch"] = subprocess.check_output(
        ["git", "branch", "--show-current"], text=True).strip()
    provenance["merge_base"] = subprocess.check_output(
        ["git", "merge-base", "HEAD", "origin/main"], text=True).strip()
    provenance["commits_ahead_of_main"] = int(subprocess.check_output(
        ["git", "rev-list", "--count", "origin/main..HEAD"], text=True).strip())
    provenance["diff_stat_vs_main"] = subprocess.check_output(
        ["git", "diff", "--stat", "origin/main...HEAD"], text=True).strip()
    # Note for forks: ensure current main is tracked (usually origin/main for personal forks).
    # If using an upstream remote, adapt the branch names to match (e.g., upstream/main).

    # === SERVER RUNTIME (MANDATORY for server tests) ===
    port = BASE_URL.split(":")[-1].rstrip("/")
    pids = subprocess.check_output(
        ["lsof", "-i", f":{port}", "-t"], text=True).strip().split("\n")
    provenance["server"] = {
        "pid": pids[0] if pids else None,
        "port": port,
        "process_cmdline": subprocess.check_output(
            ["ps", "-p", pids[0], "-o", "command="], text=True).strip() if pids else None,
        "env_vars": {var: os.environ.get(var) for var in
            ["WORLDAI_DEV_MODE", "TESTING", "GOOGLE_APPLICATION_CREDENTIALS"]}
    }

    return provenance