Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/teamspwk/nova/evaluatornpx skills add TeamSPWK/nova --skill evaluatorgit clone --depth 1 https://github.com/TeamSPWK/novaWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00063 | $0.05872 |
| Opus 5 | $0.00032 | $0.02936 |
| Sonnet 5 | $0.00013 | $0.01174 |
| Haiku 4.5 | $0.00006 | $0.00587 |
Grade D, and why
evaluator scanned grade D with 3 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Asks for rootmediumPrivilege escalation
A mod that escalates privileges can change anything on the machine, not only the project.
HIGH_RISK=$(jq -s '[.[] | select(.event_type=="tool_constraint_violation" and (.extra.matched_pattern | test("rm -rf|sudo |eval |dd if=|mkfs|chmod 777")))] | length' .nova/events.jsonl 2>/dev/null || echo 0) Recursive force deletehighDestructive command
rm -rf with a variable or a broad path is one typo away from removing the wrong tree.
HIGH_RISK=$(jq -s '[.[] | select(.event_type=="tool_constraint_violation" and (.extra.matched_pattern | test("rm -rf|sudo |eval |dd if=|mkfs|chmod 777")))] | length' .nova/events.jsonl 2>/dev/null || echo 0) Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
| **API 변경** | curl로 변경된 엔드포인트 실제 응답 확인. 상태 코드 + 응답 바디 검증 | How it starts
The opening of the file, as written. The whole thing — 394 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Nova Adversarial Evaluator
적용 규칙 (on-demand 로드)
docs/nova-rules.md §2Generator-Evaluator 분리 + 하드 게이트 (독립 서브에이전트 기술 정의)docs/nova-rules.md §3검증 기준 (기능 / 데이터 관통 / 설계 정합성 / 크래프트 / 경계값 / Coverage Gate / Learned Rules)docs/nova-rules.md §10관찰성 계약 — 판정 직후hooks/record-event.sh evaluator_verdict호출
공식 용어 매핑 (Anthropic eval framework)
Anthropic — Demystifying evals for AI agents의 5대 개념과 Nova의 매핑:
| 공식 용어 | 정의 | Nova 대응 |
|---|---|---|
| task | 입력 + 성공 기준을 갖춘 단일 테스트 | docs/nova-rules.md §3 검증 기준 (기능 / 데이터 관통 / 설계 정합성 / 크래프트 / 경계값) |
| trial | task 1회 시도 | /nova:run / /nova:check / /nova:review 한 사이클 |
| transcript (trace) | trial의 완전한 기록(출력·도구호출·중간결과) | .nova/events.jsonl (append-only JSONL) |
| outcome | 환경의 최종 상태 (≠ 에이전트 주장) | evaluator_verdict 이벤트의 verdict + 그레이딩 대상 파일 시스템 상태 |
| harness | end-to-end 실행·채점·집계 인프라 | scripts/nova-metrics.sh + hooks/record-event.sh + 본 Evaluator 서브에이전트 + .nova/events.jsonl 파이프라인 전체 |
핵심 원칙 일치:
- "outcome ≠ agent's self-report" → Nova 평가 자세 "코드가 존재하는 것과 동작하는 것은 다르다" (Layer 3 실행 검증 필수) 와 정합
- "mistakes propagate and compound across turns" → Nova "재검증 프로토콜" (수정 후 자동 재검증) 의 동기
- "trajectory metrics tell you why agents succeed or fail" → Nova
tool_constraint_violation/schema_error사후 감사 jq 쿼리가 trajectory 분석
용어 쇄신보다 병기. Nova 기존 용어(Layer 1~3·검증 기준 5종)는 그대로 유지하고, 외부 문서 크로스 레퍼런스가 필요할 때 본 매핑을 사용한다.
관찰성 훅 (v5.12.0+)
판정(PASS/CONDITIONAL/FAIL)을 내린 직후 반드시 이벤트 기록:
bash hooks/record-event.sh evaluator_verdict "$(jq -cn \
--arg v "$VERDICT" \
--argjson ci "$CRITICAL_ISSUES" \
--arg t "$TARGET" \
--arg sp "${SPRINT:-}" \
'{verdict:$v, critical_issues:$ci, target:$t, sprint:$sp}')"
VERDICT:PASS/CONDITIONAL/FAIL(대문자)TARGET:code/plan/design(소문자)- 실패는 safe-default(exit 0) — 상위 skill 영향 없음.
- Sprint 2b 이후
tool_constraint_violation이벤트를jq쿼리로 사후 감사(선언 외 도구 호출 흔적 탐지).
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago First seen · 394 lines · 63 tokens per session scan D 9e2c063689f9
evaluator is a skill published in the GitHub repository TeamSPWK/nova (2 stars, last pushed 2d ago), licensed MIT. It adds 63 tokens to every session and 5,872 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it D with 3 findings (asks for root, recursive force delete, makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
loopx-project
Use when connecting a repository or project goal document to LoopX, maintaining project-local goal state, refreshing stale dashboard status, syncing local projects into the shared global registry, or diagnosing LoopX CLI/PATH/status/history issues across multiple repos. For registering durable project materials such…
loopx-material
Operate an explicitly activated LoopX Material Lifecycle for a connected project. Use for material-store inventory, lossless migration, candidate/archive transitions, exact-read-backed ranking, ranked-entry rebuilds, bounded Explore intake, owner-gated apply, rollback, and audit. Do not use for ordinary one-off…
loopx-pr-program
Use when LoopX must manage a multi-PR or multi-MR delivery program across one or more repositories: inventory current change requests, reconcile new/merged/closed or retargeted work, preserve requirement and dependency priorities, maintain a roadmap document, or monitor material lifecycle/check/review changes over…
paperclip
Interact with the Paperclip control plane API for task coordination and governance. Use when checking assignments, updating issue status, posting comments, delegating work, managing routines, or calling Paperclip API endpoints.
wayfinder
Plan a huge chunk of work (more than one agent session can hold) as a shared map of decision tickets on your issue tracker, and resolve them one at a time until the way to the destination is clear.
github-labels-query
List GitHub repository labels with perpage pagination and name filtering support.