lai-gen-evals

Generate and Expand Evaluations

This skill guides the creation and expansion of a comprehensive evaluation suite for a Langium DSL project. A good eval suite measures not just whether an LLM can produce syntactically valid code, but whether it generates semantically correct programs that match user intent across a wide range of prompts.

You may also use the lai and langium skills for deeper understanding of the CLI workflow and Langium project structure.

When to Use

• Starter eval exists but is minimal — lai init generated a basic.eval.ts with one or two placeholder cases and you need real coverage
• No eval suite yet — the project has not been initialized; run lai init first to set up the evals directory and template files
• Eval pass rates are high but coverage is shallow — everything passes because you're only testing trivial cases
• Adding new language features — grammar or validator changes need matching eval cases
• LLM produces valid but wrong code — syntactically correct output that doesn't match what the user asked for
• Preparing for model comparison — building a thorough eval matrix to compare providers or models

Prerequisites

1. lai init completed — a lai.config.jsonc exists and an evals/ directory has been created. If not, run lai init first.
2. evals/utils.ts configured — the generateResponse() function must be wired to an LLM provider (OpenAI, Anthropic, Ollama, etc.). Check that the placeholder throw has been replaced.
3. A language descriptor and system prompt — generate these first with the lai-gen-descriptor and lai-gen-sysprompt skills if they don't exist.
4. A working Langium grammar with generated TypeScript artifacts and a create<Name>Services function.

Evaluation Architecture

File Organization

Evaluation files live in the evals/ directory (configured in lai.config.jsonc) and must have the .eval.ts extension. Organize them by evaluation category: