maf-doctor maf-testing.instructions.md

MAF Testing — Hermetic Patterns

When you write tests for MAF code in this repo, follow these patterns. They exist because we've been bitten by their absence — every entry has a "why" behind it.

Hard rules

• No live LLM calls in tests. Ever. Cost is one reason; flakiness, throttling, and "the prompt drifted in production but tests still passed" are the bigger ones.
• AAA structure with explicit comment markers: // Arrange, // Act, // Assert. Three blocks, separated by a blank line. A test that needs more than 3 blocks is doing too much.
• One assertion subject per test. Multiple Assert.X(...) calls is fine when they describe one fact (e.g., "this finding has the right ID and the right line").
• Test name follows Method_Scenario_Outcome. Example: ParseGitDiffOutput_IgnoresBinAndObjPaths_FiltersThem.
• Hermetic > integration. Prefer in-memory string sources over temp dirs. Prefer temp dirs over real network. Real network needs explicit justification.
• No Thread.Sleep, no wall-clock dependencies, no GUIDs in assertions. Each is a known source of CI flake.

The ScriptedChatClient pattern (THE canonical mock)

For any test that needs an IChatClient, never instantiate AzureOpenAIClient.AsChatClient(...). Use a scripted mock — same shape as the scaffolder generates:

private sealed class ScriptedChatClient : IChatClient
{
    private readonly string _response;
    public ScriptedChatClient(string response) => _response = response;

    public Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default)
        => Task.FromResult(new ChatResponse(new ChatMessage(ChatRole.Assistant, _response)));

    public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        yield return new ChatResponseUpdate(ChatRole.Assistant, _response);
        await Task.CompletedTask;
    }

    public object? GetService(Type serviceType, object? serviceKey = null) => null;
    public void Dispose() { }
}