mcp-server-ppt llm-testing-philosophy.instructions.md

LLM Testing Philosophy

⚠️ CORE PRINCIPLE: Tests simulate real users. Failures expose product gaps, not test gaps.

What Are LLM Tests?

LLM tests use an AI coding agent (with its own default system prompt) to exercise our CLI and MCP tools. The agent receives a natural language prompt — the same kind a real user would type — and must figure out how to accomplish the task using our tools.

These tests do NOT test the LLM. They test our product's usability surface:

• Skill documentation (SKILL.md)
• CLI --help output
• MCP tool descriptions (XML /// <summary>)
• Error messages and recovery hints
• Parameter naming and discoverability
• Workflow coherence

The Golden Rule

If the LLM can't figure it out, fix the product — never fix the test.

When an LLM test fails, the root cause is ALWAYS one of:

1. Our skill docs don't explain the workflow clearly enough
2. Our tool descriptions are misleading or incomplete
3. Our CLI --help output doesn't show the right examples
4. Our error messages don't guide recovery
5. Our parameter names are confusing or undiscoverable
6. The test itself is unreasonable (rare — only fix if a human couldn't do it either)

What NEVER Belongs in a Test

❌ xfail or skip Markers

NEVER use @pytest.mark.xfail or @pytest.mark.skip to hide failing tests.

Tests either pass or fail. There is no middle ground.

• xfail masks real failures and creates a false sense of progress
• skip hides broken code instead of fixing it
• If a test fails, fix the product — the test is exposing a real problem (Golden Rule)
• If a test is flaky, fix the flakiness — don't paper over it with xfail

❌ CLI Command Guidance in Prompts

A real user doesn't know our CLI syntax. Neither should the test prompt.

# ❌ WRONG: Teaching the LLM how to use our CLI
prompt = """
Create a slide layout then set it to use a custom master using 
'pptcli slidemaster' (run --help to see options).
The custom layout uses layout-type value 0.
"""

# ✅ CORRECT: Natural user request
prompt = """
Create a presentation with a custom slide layout showing
title and content areas.
"""