generate-synthetic-data

generate-synthetic-data is a skill for Claude Code, Codex from marchatton/agent-skills. It costs 74 tokens per session (1,184 once invoked), scanned A, a copy of generate-synthetic-data, MIT.

A guide for creating artificial test inputs for evaluating an LLM pipeline. It varies realistic factors such as user type, task, and scenario so the tests cover likely failure areas.

In plain words
What is it for?
Use it to bootstrap an LLM evaluation dataset or stress-test particular failure hypotheses before running the pipeline.
Why use it?
It helps build an evaluation dataset when real user examples are scarce or when specific failure cases need testing.

Skill for Claude CodeCodex

Written for no agent in particular: nothing here depends on one. Also seen: installed under .agents/ (shared by several agents).

Good fit Use it to bootstrap an LLM evaluation dataset or stress-test particular failure hypotheses before running the pipeline.

Compare 6 skills from other repositories ↓
Install with agentmods
npx agentmods add skills/marchatton/agent-skills/generate-synthetic-data
Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent
npx skills add marchatton/agent-skills --skill generate-synthetic-data
Clone the repo
git clone --depth 1 https://github.com/marchatton/agent-skills

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for generate-synthetic-data

README.md
[![agentmods](https://agentmods.dev/badge/skills/marchatton/agent-skills/generate-synthetic-data/github.svg)](https://agentmods.dev/skills/marchatton/agent-skills/generate-synthetic-data)
Your own site
<a href="https://agentmods.dev/skills/marchatton/agent-skills/generate-synthetic-data"><img src="https://agentmods.dev/badge/skills/marchatton/agent-skills/generate-synthetic-data/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

agentmods 80×15 button for generate-synthetic-data

Your own site · 80×15
<a href="https://agentmods.dev/skills/marchatton/agent-skills/generate-synthetic-data"><img src="https://agentmods.dev/badge/skills/marchatton/agent-skills/generate-synthetic-data.svg" alt="Reviewed on agentmods" width="80" height="20"></a>
Per session 74 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 1,184 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.
Origin 100% copy Near-identical to another mod in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5.1 $0.00074 $0.01184
Opus 5 $0.00037 $0.00592
Sonnet 5 $0.00015 $0.00237
Haiku 4.5 $0.00007 $0.00118

Measured 7d ago against content hash 37b8e12ceb02, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-10, from the pricing page.

Security

Grade A, and why

generate-synthetic-data scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 7d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

Origin

This is a copy

100% identical to generate-synthetic-data — 0 lines differ, which has more behind it and is treated as the original. This page carries a canonical link to it rather than competing with it.

.agents/skills/08-evals/generate-synthetic-data/SKILL.md · 132 lines

How it starts

The opening of the file, as written. The whole thing — 132 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Generate Synthetic Data

Generate diverse, realistic test inputs that cover the failure space of an LLM pipeline.

Prerequisites

Before generating synthetic data, identify where the pipeline is likely to fail. Ask the user about known failure-prone areas, review existing user feedback, or form hypotheses from available traces. Dimensions (Step 1) must target anticipated failures, not arbitrary variation.

Core Process

Step 1: Define Dimensions

Dimensions are axes of variation specific to your application. Choose dimensions based on where you expect failures.

Dimension 1: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Dimension 2: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Dimension 3: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Example for a real estate assistant:

Feature: what task the user wants
  Values: [property search, scheduling, email drafting]

Client Persona: who the user serves
  Values: [first-time buyer, investor, luxury buyer]

Scenario Type: query clarity
  Values: [well-specified, ambiguous, out-of-scope]

Start with 3 dimensions. Add more only if initial traces reveal failure patterns along new axes.

Step 2: Draft 20 Tuples with the User

A tuple is one combination of dimension values defining a specific test case. Present 20 draft tuples to the user and iterate until they confirm the tuples reflect realistic scenarios. The user's domain knowledge is essential here — they know which combinations actually occur and which are unrealistic.

(Feature: Property Search, Persona: Investor, Scenario: Ambiguous)
(Feature: Scheduling, Persona: First-time Buyer, Scenario: Well-specified)
(Feature: Email Drafting, Persona: Luxury Buyer, Scenario: Out-of-scope)

Step 3: Generate More Tuples with an LLM

Generate 10 random combinations of ({dim1}, {dim2}, {dim3})
for a {your application description}.

The dimensions are:
{dim1}: {description}. Possible values: {values}
{dim2}: {description}. Possible values: {values}
{dim3}: {description}. Possible values: {values}

Output each tuple in the format: ({dim1}, {dim2}, {dim3})
Avoid duplicates. Vary values across dimensions.

Read the full file on GitHub · 132 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 7d ago First seen · 132 lines · 74 tokens per session scan A 37b8e12ceb02

Subscribe to this mod's changes

generate-synthetic-data is a skill published in the GitHub repository marchatton/agent-skills (5 stars, last pushed 6mo ago), licensed MIT. It adds 74 tokens to every session and 1,184 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. It is 100% identical to generate-synthetic-data, differing in 0 lines, and is treated as a copy.

Related

Other skills, from other repositories

cli-eval

Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.

diegosouzapw/OmniRoute · 34 tokens

model-merging

Merge multiple fine-tuned models using mergekit to combine capabilities without retraining. Use when creating specialized models by blending domain-specific expertise (math + coding + chat), improving performance beyond single models, or experimenting rapidly with model variants. Covers SLERP, TIES-Merging, DARE, Task…

davila7/claude-code-templates · 73 tokens

darwinian-evolver

Evolve prompts/regex/SQL/code with Imbue's evolution loop.

NousResearch/hermes-agent · 22 tokens

validate

Validate Semantica pipelines, extraction quality, graph schemas, and ontology consistency. Returns structured error/warning checklists. Uses PipelineValidator, PipelineBuilder.validatepipeline(), GraphValidator, and OntologyValidator. Sub-commands: pipeline, step, dependencies, extraction, graph, ontology, performance.

semantica-agi/semantica · 0 tokens

launching-evals

Run, monitor, analyze, and debug LLM evaluations via nemo-evaluator-launcher. Covers running evaluations, checking status and live progress, debugging failed runs, exporting artifacts and logs, and analyzing results. ALWAYS triggers on mentions of running evaluations, checking progress, debugging failed evals…

NVIDIA/Model-Optimizer · 115 tokens

nemo-automodel-recipe-development

Create and modify NeMo AutoModel training and evaluation recipes, including YAML structure, builders, and execution flow.

NVIDIA/skills · 31 tokens