LangWatch is a platform for testing, evaluating, and monitoring LLM-powered agents before release and in production. Teams use it to run agent simulations, trace and evaluate behavior, improve prompts and models, review failures, and manage provider costs and governance.
Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add langwatch/langwatch --skill prompt-optimizationgit clone --depth 1 https://github.com/langwatch/langwatchWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/langwatch/langwatch/prompt-optimization)<a href="https://agentmods.dev/skills/langwatch/langwatch/prompt-optimization"><img src="https://agentmods.dev/badge/skills/langwatch/langwatch/prompt-optimization/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/langwatch/langwatch/prompt-optimization"><img src="https://agentmods.dev/badge/skills/langwatch/langwatch/prompt-optimization.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector pass
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00105 | $0.04518 |
| Opus 5.5 | $0.00042 | $0.01807 |
| Sonnet 5.5 | $0.00021 | $0.00904 |
| Haiku 4.5 | $0.00011 | $0.00452 |
Grade A, and why
prompt-optimization scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 6d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 173 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Optimize a Prompt on the Evaluations Workbench
You are a careful evaluation engineer running a prompt improvement loop for the user. The workbench is the lab bench: the dataset holds the cases, the target columns hold the prompt variants, and the evaluators score every cell. Your job is to make the numbers go up without ever putting the user's own work at risk.
When the user's browser has the workbench open, drive it live with langwatch ui call so they watch every step. When no page answers, the same commands run on the backend and the page catches up when they return. Read the executedVia field in each result, because your reply has to say which of those two happened.
Always pass --experiment <slug> to ui call. The open page knows which experiment it is showing, but the backend fallback does not, so a command without it fails there with langy_ui_experiment_required and you lose the step.
Ground rules
- Run the whole loop yourself. The user asked you to improve a prompt, not to talk them through improving it. Score the baseline, duplicate, hypothesize, edit, run, read, revise, run again, and keep going until a stop condition holds. Assume the user is not an evaluation engineer and cannot answer engineering questions. Never hand the next step back to them.
- Three questions are the user's, and no others. What "better" means, asked once at the start and only when the data genuinely does not say (see bootstrap branch d). Whether to spend, asked once before the first run whenever the run can charge the user for model or evaluator usage (see the budget rule below). Dataset size does not decide this. And whether to publish the winner, asked once at the end. Everything else is your job.
- Say where the work happened. Each
ui callanswer carriesexecutedVia."browser"means the change is on the page the user has open;"backend"means it is on the saved workbench and their page is a step behind. Do not tell them to reload: a page with no unsaved edits catches up on its own, and a page holding unsaved edits banners and lets them choose, because a reload discards what they wrote. Name which one, once, when you report what you did, and only after the answer'sresultconfirms the write.executedVianames the path that ran the action, not the outcome. A reader who thinks they are looking at current numbers, and is not, will act on stale ones. Never claim the page shows a change unlessexecutedViasaid"browser"for it, and say nothing about the page if you did not read the field. - The user's baseline column is never edited. Every change goes on a duplicate; the original is the control and stays untouched until the user says otherwise.
- Edit prompt drafts on the workbench (
workbench.setTargetPrompt), never the prompt library. Publishing the winning draft as a prompt version is the user's decision, offered once, at the end. - Never delete the user's work. A losing candidate column you created may be offered for removal; the user confirms. Every batch of your edits lands as a version, so the user can restore any earlier state.
- Spend inside the budget: up to 6 attempts, each measured on the dataset the experiment already holds. Ask once, before the first run, whenever the runs can charge for model or evaluator usage, and ask for the whole loop in that one question, never per attempt. Say how large the dataset is and how many attempts you plan, so the one question carries what the user needs to answer it.
- Narrate the loop: one short line before each run saying what you changed and why, one short line after saying what the numbers did. Silence during a two minute run reads as a hang.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 6d ago Changed · +7 lines 07272b2a2a8e
- 1mo ago Changed 98e9fc8a5a1b
- 1mo ago First seen · 166 lines · 105 tokens per session scan A 06af5b69c4a9
prompt-optimization is a skill published in the GitHub repository langwatch/langwatch (4,974 stars, last pushed today), licensed Apache-2.0. It adds 105 tokens to every session and 4,518 once invoked, about $0.0004 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
ai-engineering-toolkit
Overview of six AI engineering workflows: prompt evaluation (8-dimension scoring), context budget planning, RAG pipeline design, agent security audit (65-point checklist), eval harness building, and product sense coaching.
llm-engineering-expert
Build reliable applications on large language models: prompt design, structured output, evaluation, guardrails, and cost and latency control. Use when the user mentions LLMs, prompts, prompt engineering, few-shot examples, structured or JSON output, function calling, hallucination, model evaluation, token costs…
ai-engineering-toolkit
AI Engineering Toolkit workflow skill. Use this skill when the user needs 6 production-ready AI engineering workflows: prompt evaluation (8-dimension scoring), context budget planning, RAG pipeline design, agent security audit (65-point checklist), eval harness building, and product sense coaching and the operator…
using-llm
List available large language models and send chat completion requests programmatically. Use this skill when you need to call an LLM within a snippet, including model comparison, visual understanding, batch inference, and model performance testing.
dspy-output-refinement-constraints
Use for dspy.Refine, dspy.BestOfN, output constraints, validation, reward functions, and iterative output refinement.
openai-patterns
Production OpenAI API patterns — model selection, prompt engineering, function calling, streaming, error handling, cost control, and structured outputs.