canary-add-framework

Canary: Add Framework

Add a new testing framework to Canary's registry end-to-end. Confirms the classifier↔registry contract, authors the registry entry, validates the execution command, and ensures every classifier test_type still resolves to a non-null framework.

When to Use

• When the user asks to add support for a new framework ("add k6", "support cypress", "we need Locust")
• When a classifier test_type exists but has no framework backing it (registry gap)
• When migrating from a deprecated framework — adding the new entry before removing the old one
• NOT for choosing between existing frameworks at runtime — that's the recommender's job
• NOT for adding a one-off LLM-generated framework name that doesn't exist as a real tool — frameworks must be real, runnable CLIs

Process

Phase 1: SCOPE — Confirm the Framework Belongs in the Registry

1. Verify the framework is real and installable. It needs a runnable CLI invocation (the execution_command template). If you can't write a one-line shell command that runs a single test file, it doesn't fit Canary's model.
2. Identify the category. Match to an existing category value (e2e_ui, python_unit, api, performance, frontend_unit, etc.). New categories require a coordinated classifier update — see Phase 2.
3. Check for overlap. Is there already a preferred framework in this category? If yes, decide intentionally: is the new framework a replacement (demote the old to legacy), a peer (add as supported), or the new preference (demote the old, mark new as preferred)?
4. Confirm the user wants this maintained. Each registry entry is a maintenance commitment. A framework added casually but never validated rots into a routing trap.

Phase 2: ALIGN — Classifier ↔ Registry Contract

1. Find the classifier rule that emits this test_type. Open ts/src/core/classifier.ts and locate the heuristic block that produces the target category.
2. If no classifier rule exists for the category: Add one before the registry entry. A framework with no routing path is a dead entry. Add a heuristic that maps natural-language signals (keywords, phrases) to the new test_type.
3. If the category exists but the classifier never emits it confidently: Strengthen the rule's heuristics. Confidence below 0.7 should trigger a clarification, not a generation.
4. Run the contract tests: