fully-automated-prompt-optimization: Command for Claude Code

---

description: > Create realistic synthetic examples for eval dataset augmentation. TRIGGER when: user wants to create synthetic test cases, add edge cases, augment eval datasets, expand test coverage, create hard cases, or generate new evaluation examples. DO NOT TRIGGER when: user is pruning/cleaning existing synthetic data (use synthetic-pruner), running evals (use eval-runner), or optimizing prompts (use optimization agent).

Synthetic Samples

Scope

• Create realistic synthetic examples under a tenant synthetic examples root (for example, tenants/<tenant_id>/datasets/synthetic_artifacts/).
• Keep customer/source artifacts untouched (tenants/*/source_artifacts/).
• Produce review CSVs for model-assisted labeling.
• Keep all content synthetic and non-attributable. No real customer names, domains, or IPs.

Quick Start Workflow

1. Pick a scenario type: benign, phishing, credential theft, BEC, malware delivery, or false positive.
2. Choose a naming pattern and create the example directory.
3. Populate context files using the templates below.
4. Write Summary.pdf.txt with explicit labels that match CSV heuristics.
5. Add/refresh labels_review.csv or hard_labels_review.csv entries.

Standard Example Structure

Each example directory should include:

• Prompt.pdf.txt (placeholder or template reference)
• Context - Tools.pdf.txt (copy from a tenant baseline example if available)
• Context - Stealth Watch.pdf.txt
• Context - Email Body.pdf.txt or Context - No Email Body.pdf.txt
• Summary.pdf.txt (explicit label language)

Naming

• Regular: Example N - <Descriptor>
• Hard cases: Hard Example N - <Descriptor>

Labeling Workflow

1. Propose labels in a CSV (labels_review.csv or hard_labels_review.csv).
2. Keep labels aligned with Summary.pdf.txt text so existing heuristics extract GT.
3. If a file is removed, update both CSVs so row counts and filenames stay aligned.