live-tests

Live Test & Fixture Recording Skill

You write live integration tests that exercise the real GitHub Copilot API and capture responses as JSON fixtures for deterministic replay. This is the project's VCR-like system for ensuring tests stay grounded in real API behavior.

Architecture overview

Live test (mix test --include live --include save_fixtures)
  │
  ├─ RecordingProvider  ← wraps real Copilot provider, captures SSE events
  │     │
  │     └─ persistent_term storage ← events buffered here during stream
  │
  └─ FixtureHelper.save_fixture/2  ← writes events to JSON file
        │
        └─ test/support/fixtures/<name>.json

Fixture-based test (mix test)
  │
  ├─ FixtureProvider  ← reads fixture JSON, replays events via Req.Response.Async
  │     │
  │     └─ FixtureHelper.build_fixture_response/1 ← spawns process to send SSE events
  │
  └─ Assertions on agent behavior, events, tool calls, etc.

When to act

• Adding a new agent behavior that needs a new fixture (e.g., new tool call pattern, multi-turn conversation, error handling).
• Adding or changing provider integration logic.
• When the user asks to "record a fixture", "add a live test", or "capture API responses".
• When an existing fixture is stale and needs re-recording.

Writing a live test

Live tests go in test/opal/live_test.exs. They require @moduletag :live (excluded by default) and use the RecordingProvider to capture real SSE events.

Template

describe "live API — <scenario description>" do
  @tag :save_fixtures
  @tag timeout: 30_000
  test "records <what this captures>" do
    RecordingProvider.start_recording()

    {:ok, pid} =
      Opal.start_session(%{
        model: {:copilot, "claude-sonnet-4"},
        system_prompt: "<constrained prompt that produces deterministic output>",
        tools: [<tool modules if needed>],
        working_dir: System.tmp_dir!(),
        provider: RecordingProvider
      })

    {:ok, response} = Opal.prompt_sync(pid, "<user message>", 25_000)
    Opal.stop_session(pid)

    events = RecordingProvider.stop_recording()
    assert length(events) > 0

    # Save as a fixture for replay
    path = FixtureHelper.save_fixture("<descriptive_name>.json", events)
    assert File.exists?(path)

    # Assertions on the live response
    assert is_binary(response)
    assert String.contains?(response, "<expected content>")
  end
end