huggingface-community-evals

Overview

This skill is for running evaluations against models on the Hugging Face Hub on local hardware.

It covers:

• inspect-ai with local inference
• lighteval with local inference
• choosing between vllm, Hugging Face Transformers, and accelerate
• smoke tests, task selection, and backend fallback strategy

It does not cover:

• Hugging Face Jobs orchestration
• model-card or model-index edits
• README table extraction
• Artificial Analysis imports
• .eval_results generation or publishing
• PR creation or community-evals automation

If the user wants to run the same eval remotely on Hugging Face Jobs, submit the same script via hf jobs uv run (CLI) or the hf_jobs() MCP tool if configured, for remote GPU execution.

If the user wants to publish results into the community evals workflow, stop after generating the evaluation run and hand that publishing step off to the project's own community-evals workflow.

All paths below are relative to the directory containing this SKILL.md.

When To Use Which Script

Prerequisites

• Prefer uv run for local execution.
• Set HF_TOKEN for gated/private models.
• For local GPU runs, verify GPU access before starting:

uv --version
printenv HF_TOKEN >/dev/null
nvidia-smi

If nvidia-smi is unavailable, either:

• use scripts/inspect_eval_uv.py for lighter provider-backed evaluation, or
• submit it to Hugging Face Jobs (hf jobs uv run or hf_jobs()) if the user wants remote compute.

Core Workflow

1. Choose the evaluation framework.
   • Use inspect-ai when you want explicit task control and inspect-native flows.
   • Use lighteval when the benchmark is naturally expressed as a lighteval task string, especially leaderboard-style tasks.
2. Choose the inference backend.
   • Prefer vllm for throughput on supported architectures.
   • Use Hugging Face Transformers (--backend hf) or accelerate as compatibility fallbacks.
3. Start with a smoke test.
   • inspect-ai: add --limit 10 or similar.
   • lighteval: add --max-samples 10.
4. Scale up only after the smoke test passes.
5. If the user wants remote execution, submit the same script + args to Hugging Face Jobs (hf jobs uv run or hf_jobs()).