release-eval-report

Release eval report

Use this skill only for a validated automated release-eval event. Produce one short Slack root message and exactly three detailed thread replies. The report should help the team decide which evals are saturated enough to become regression checks and which evals need auditing.

Separate the root and each reply with this exact line:

<!-- RELEASE_EVAL_THREAD_REPLY -->

The renderer removes the separators and posts the four parts as one root message and three replies. Never use the separator for any other purpose.

Root message

Match the established reports in #next-bots: a short framing paragraph, followed by a few named findings written in compact prose.

• Use only Next.js <version> as the title.
• In the opening paragraph, identify release, backport, or canary, matrix size, and the Base and AGENTS.md totals.
• Organize the findings around the questions the team is answering: Saturation, Harder tasks, Potential eval problems, and Comparison.
• Saturation should quantify tasks above 90% in both variants, identify the strongest regression-suite candidates, and say whether the strongest agents are already saturated.
• Harder tasks should summarize the tasks below 90%, say whether failures cluster among weaker agents, and call out exceptions where a high-scoring agent failed.
• Potential eval problems should flag extreme results such as 0% tasks or surprising confirmed failures. Describe them as audit candidates unless the report data proves a cause.
• Comparison should summarize changed tasks, changed agent totals, and agent version changes only when a comparable prior run exists. On the first run, say that it establishes the controlled baseline and omit deltas.
• Do not include execution links, pull requests, artifacts, metric methodology, or automation commentary.
• Do not use tables. Keep the root under 2,500 characters.

Use this shape:

**Next.js <version>**

This <release|backport|canary> measured <task count> tasks across <agent count> agents. Without AGENTS.md, <passed>/<total> passed (<percent>%); with AGENTS.md, <passed>/<total> passed (<percent>%).

**Saturation**
<count and interpretation of tasks above 90%, plus top-agent saturation>

**Harder tasks**
<count and interpretation of tasks below 90%, failure clusters, and notable exceptions>

**Potential eval problems**
<extreme or surprising results that warrant auditing, or `None`>

**Comparison**
<changes from the previous comparable run, or first-run baseline note>