bench-contribute

/bench-contribute

Turn your real corrections into benchmark scenarios that help measure whether AI agents actually learn from feedback. Your corrections are the most valuable training data because they're authentic — they came from real drift in real sessions.

This runs inside the agent, not the shell. The agent reads your feedback files and generates anonymized scenarios with your approval.

Steps

1. Read all feedback files from ~/.skopus/memory/feedback/*.md. For each file, extract:

   • title — what the rule is
   • why — why it matters
   • how_to_apply — when and where it kicks in

2. Check which corrections are already in the dataset. Read ~/skopus/bench/correction_persistence/dataset.json (or the installed package's copy). Skip any correction that already has a matching scenario (by charter_relevance field).

3. For each NEW correction, generate a candidate scenario with these fields:

   {
     "id": "cp-NNN",
     "domain": "code | prose | reasoning | tool-use",
     "title": "<short title>",
     "initial_task": "<a realistic task where the agent would make this mistake>",
     "expected_mistake_pattern": "<what the agent typically gets wrong>",
     "correction": "<the correction, generalized to remove proprietary details>",
     "followup_task": "<a SIMILAR but DIFFERENT task to test if the correction persists>",
     "success_criterion": {
       "must_include": ["<keywords that indicate the correction was applied>"],
       "must_not_include": ["<keywords that indicate the mistake was repeated>"]
     },
     "charter_relevance": "<which charter non-negotiable this tests>"
   }
   

4. Anonymize and generalize. Remove:

   • Company names, product names, internal tool names
   • Specific file paths, class names, variable names
   • Personal names, team member names
   • API keys, URLs, internal endpoints

   Replace with generic equivalents. "Don't use npm in the UEVA frontend Docker setup" becomes "Don't suggest local npm when the project uses Docker containers for the frontend."