nasde-benchmark-creator

NASDE Benchmark Creator

Create and configure coding agent benchmarks for evaluation with nasde. A benchmark is a set of coding tasks that AI agents solve inside isolated Docker containers, scored both by functional tests (pass/fail) and by an LLM-as-a-Judge architecture assessment.

Critical: line endings on Windows (read this first)

Benchmark scripts execute inside Linux sandboxes (Docker, Daytona). If tests/test.sh, solution/solve.sh, or environment/Dockerfile are checked out with CRLF line endings (the Windows git default when core.autocrlf=true and there is no .gitattributes), every trial fails immediately with:

bash: line 1: /tests/test.sh: cannot execute: required file not found

…because the kernel reads the shebang as #!/bin/bash\r and tries to execute a non-existent /bin/bash\r. The agent finishes its work, but the verifier never runs and Harbor reports RewardFileNotFoundError.

Mitigation (always do this for a new benchmark — nasde init does it for you, but verify):

1. The benchmark repo MUST have a .gitattributes file enforcing LF for shell scripts and Dockerfiles. The minimum content:

   * text=auto eol=lf
   *.sh        text eol=lf
   *.bash      text eol=lf
   Dockerfile  text eol=lf
   *.dockerfile text eol=lf
   docker-compose.yaml text eol=lf
   docker-compose.yml  text eol=lf
   
   *.ps1       text eol=crlf
   *.bat       text eol=crlf
   *.cmd       text eol=crlf
   

   nasde init writes this automatically. If you are adding a benchmark to an existing repo without .gitattributes, create one before adding any task.

2. When writing .sh or Dockerfile content programmatically on Windows, write with explicit LF — not path.write_text(content) (which translates \n→\r\n on Windows), but path.write_text(content, encoding="utf-8", newline="") or open the file in binary mode.

3. After committing on Windows for the first time, run:

   git add --renormalize .
   git commit -m "normalize line endings"
   

   to fix any files that landed before .gitattributes was in place.