software-environment-containerization-setup

software-environment-containerization-setup

Summary

Create and verify a reproducible Conda environment for Hi-C data processing pipelines, ensuring all Python (>3.7), R, and compiled tool dependencies are correctly installed, resolved in PATH, and functionally validated. This skill bridges the gap between abstract dependency specifications and a working, testable runtime.

When to use

You have a bioinformatics pipeline (like HiC-Pro) with mixed Python, R, and compiled tool dependencies, and you need to ensure consistent reproducibility across machines and team members without manual per-tool installation. Use this when dependencies include version-pinned libraries (e.g., scipy >=1.4.1, ggplot2 >2.2.1), optional auto-installable tools (bowtie2, samtools), and external modules no longer bundled with the pipeline (iced).

When NOT to use

• The pipeline is already running successfully in a pre-built Docker or Singularity container and no local environment customization is needed.
• All dependencies are already system-installed (non-Conda) and you only need to point to existing paths via config files, not create a new isolated environment.
• You are working in a cluster scheduler (TORQUE, SGE, SLURM, LSF) where module files or pre-configured environments are centrally managed and no custom Conda environment is permitted.

Inputs

• environment.yml or conda environment specification file
• Miniconda/Anaconda installation (or system Python with conda available)
• Optional: config-install.txt or similar configuration template for tool path overrides

Outputs

• Activated Conda environment with all dependencies installed and verified
• Dependency verification report documenting Python version, all installed library versions, R packages, tool paths, and import/execution test results
• Summary of resolved tool paths (bowtie2, samtools, iced) in PATH or environment variables

How to apply

Start by obtaining or reconstructing the environment specification file (environment.yml) that lists all Conda-resolvable dependencies with version constraints. Install Miniconda if absent, then create a new isolated Conda environment using conda env create -f environment.yml -p <install_path>. Activate the environment and systematically verify each dependency class: (1) Python version and importability of core libraries (bx-python >=0.8.8, numpy >=1.18.1, scipy >=1.4.1, pysam >=0.15.4) via import statements; (2) R availability and required packages (ggplot2 >2.2.1, RColorBrewer, grid) via library() calls in R; (3) compiled tool binaries (bowtie2, samtools >=1.9) in PATH via which and --version checks; (4) separately install and verify non-bundled modules (iced from https://github.com/hiclib/iced). Document all resolved paths, versions, and import success in a summary report, which serves as proof of correct containerization and enables debugging if downstream tools fail.