data-verify

Data-Verify Skill

Verify data components produced by paper-filter and data-check. Outputs a verification report (data_verify_result.json) for human review — does not modify filter_result.json or data directories.

Context

paper-filter (Level 1-3) and data-check (Level 4) run automatically and may produce systematic errors in data component classification, evaluation setting identification, and separability descriptions. This skill performs independent verification and outputs an actionable report.

Core Principle

Provide Algorithm A with the same initial conditions it had, but do not include any of A's own operations or outputs.

Why: The goal is not to reproduce the paper's algorithm — it is to test whether a Solver can independently solve the same ML problem using any method, potentially surpassing the original. The benchmark gives the Solver the same starting point as the paper's authors, then scores its output against ground truth using the paper's metrics, with the paper's reported scores as baselines for comparison. Including A's outputs would leak solutions; excluding legitimate initial data would make the task unfairly harder.

Data Component Classification

See the classification table in references/check_rules.md § Terminology Reference for the six categories (initial_state, data_preparation, algorithm_preprocessing_output, algorithm_output, external_resource, irrelevant) and their definitions.

Input Requirements

Before invoking this skill, provide:

1. Paper Folder Path: Directory containing paper files and data-check results

Paper folder must contain:

• filter_result.json (with final_result.data_check_passed == true)
• repositories/ (cloned code repositories from data-check)
• data/ (acquired data from data-check, organized per setting)
• preprocessed/ (paper-preprocess output: text.md, figures/, tables/, links.json)
• {paper_id}.pdf and {paper_id}.html (original paper, for additional reference)