ci

Wire coder-eval into GitHub Actions

The user's request is: $ARGUMENTS

Step 1 — Check the repository

Find the repository's task tree by following ${CLAUDE_PLUGIN_ROOT}/reference/repo-layout.md, and check whether .github/workflows/ exists. The paths you resolve here become args: entries in the workflow in step 3 — that input is written from discovery, never from a fixed guess.

If there is no .github/ directory at all, say that this skill targets GitHub Actions and stop — do not invent an equivalent for another CI system unless the user asks.

If a workflow already runs coder-eval (grep the workflows for coder_eval), do not add a second one. Show what is there and offer to update it.

Step 2 — Choose the trigger

Ask, or infer from the request:

• On pull request — gate changes to the tasks or to whatever they exercise.
• On a schedule — the skill-drift case: re-run the suite weekly against the current model so a skill that quietly stops triggering surfaces before users hit it. This is the trigger most repositories actually want, and the one they forget. If the suite is an activation suite, the environment note in step 3 is not optional for this trigger — without it the scheduled run reports total drift every week regardless of whether anything drifted.
• Both, which is fine — one workflow, two on: keys.

Step 3 — Emit the workflow

The composite action installs the coder-eval CLI and nothing else: it is agent-agnostic and installs no coding-agent runtime. A task using the default claude-code agent therefore needs Node plus the Claude CLI provided by the job first, or the run dies on a missing claude binary. There is no Marketplace install step for the action itself, but those two prerequisite steps are not optional.

name: Coder Eval

on:
  pull_request:
  schedule:
    - cron: "0 6 * * 1"   # Mondays 06:00 UTC — catches model/skill drift

# Least privilege: this job runs agent-generated code, so it gets no write scope.
permissions:
  contents: read

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v6
        with:
          # Do not leave a credentialed .git/config in a workspace where
          # agent-generated code runs.
          persist-credentials: false

      # The action installs no coding-agent runtime — provide it here.
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
      - run: npm install -g @anthropic-ai/claude-code

      - id: eval
        uses: UiPath/coder_eval@v0
        with:
          run-dir: runs/ci
          args: |
            tasks/**/*.yaml
            --model
            claude-haiku-4-5-20251001
          env: |
            ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}

      - if: always()
        run: cat "${{ steps.eval.outputs.run-md-path }}" >> "$GITHUB_STEP_SUMMARY"