task-build

Task Build Skill

Build a structured ML benchmark task package from a CNS paper that has completed the three preprocessing stages (paper-preprocess → paper-filter → data-check).

Input Requirements

Before invoking this skill, provide:

1. Paper Folder Path: Directory containing the paper and all prior processing results
2. Output Directory: Where to build the final task package (can be same as paper folder or a new location)

Paper folder must contain:

• {paper_id}.pdf and {paper_id}.html: Original paper files
• preprocessed/: paper-preprocess output (text.md, links.json, figures/, tables/)
• filter_result.json: Combined output from paper-filter + data-check (must have final_result.data_check_passed == true)
• data/: Acquired data organized per setting (data-check output)
• repositories/: Cloned repositories (data-check output, reference only)

Paper folder may also contain:

• data_verify_result.json (optional): Independent verification report from data-verify. When present, use algorithm_boundary.file_classifications as supplementary reference for file role judgment (e.g., distinguishing algorithm artifacts from initial state, identifying external resources/oracles). When it conflicts with filter_result.json, verify against actual repository code and data to determine the correct classification.

Output Structure

{output_dir}/
├── problem/                        # Solver-visible package
│   ├── data/                       # Solver-visible data
│   │   ├── {setting_1}/            # One directory per evaluation setting
│   │   │   ├── [d_dev files]       # Training data, validation data, pretrained models, etc.
│   │   │   └── [x_test files]      # Test inputs
│   │   └── {setting_2}/
│   │       └── ...
│   ├── data_description.md         # Technical data documentation
│   └── README.md                   # Task definition document
├── evaluation/                     # Evaluator package (hidden from solver)
│   ├── evaluator.py                # Automated evaluation script (reads from workspace/output/)
│   └── ground_truth/               # Reference answers
│       ├── {setting_1}/
│       │   └── [y_ref files]
│       └── {setting_2}/
│           └── [y_ref files]
├── environment/                    # Execution environment
│   ├── Dockerfile.v3               # Task-specific Docker image definition
│   └── packages.json               # Package manifest for automated verification
└── metadata.json                   # Task metadata and performance baselines