ai-evals-course

ai-evals-course/evals-skills

Plugin Claude Code

Plugin marketplace listing 1 plugin: evals.

ai-evals-course/evals-skills

Plugin Claude Code

Skills for building LLM evals. Starts with interactive error discovery: build a review app, sample diverse traces, and organize human annotations into failure modes.

ai-evals-course/evals-skills

Skill Claude CodeCodex

Build a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedback. Use when you need to build an annotation tool, review traces, or collect human labels.

ai-evals-course/evals-skills

Skill Claude CodeCodex

Run error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes.

ai-evals-course/evals-skills

Skill Claude CodeCodex

Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch…

ai-evals-course/evals-skills

Skill Claude CodeCodex

Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.

ai-evals-course/evals-skills

Skill Claude CodeCodex

Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation. Use when bootstrapping an eval dataset, when real user data is sparse, or when stress-testing specific failure hypotheses. Do NOT use when you already have 100+ representative real traces (use stratified sampling…

ai-evals-course/evals-skills

Skill Claude CodeCodex

Entry point for evals. Use when the user asks for help with evals, does not know where to begin, or asks for something no other skill in this plugin matches. Do NOT use when a more specific skill in this plugin already matches; load that skill directly.

ai-evals-course/evals-skills

Skill Claude CodeCodex

Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (write-judge-prompt) when you need to verify alignment before trusting its outputs. Do NOT use for code-based evaluators (those are deterministic; test with standard unit tests).

ai-evals-course/evals-skills

Skill Claude CodeCodex

Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle. Use when a failure mode requires interpretation (tone, faithfulness, relevance, completeness). Do NOT use when the failure mode can be checked with code (regex, schema validation, execution tests). Do NOT use when you need to…

ai-evals-course/isaac-fasthtml-workshop

Cursor rule Cursor

Rules for working with the React project in the /backend folder.

ai-evals-course/isaac-fasthtml-workshop

Cursor rule Cursor

Rules for working with the React project in the /frontend folder.

ai-evals-course/isaac-fasthtml-workshop

Cursor rule Cursor

The main file for the project is mainsqlite.py It uses MonsterUI for styling. MonsterUI is a python first UI component library that primarily leverages FrankenUI and Tailwind, but also includes headers and functionality form DaisyUI, Katex, HighlightJS, and others.

ai-evals-course/isaac-fasthtml-workshop

Cursor rule Cursor

This rule is to help create interactive behavior with HTMX and fasthtml, which is the preferred approach.

ai-evals-course/isaac-fasthtml-workshop

Cursor rule Cursor

Rules for implementing common UI elements in MonsterUI idiomatically.