ngs-pipeline-management

NGS Pipeline Management

Overview

NGS pipeline management involves building reproducible, scalable workflows for processing next-generation sequencing data. This skill covers Snakemake and Nextflow for workflow definition, execution on HPC clusters and cloud platforms, containerization with Docker/Singularity, and best practices for reproducible bioinformatics.

When to Use This Skill

• Building RNA-seq, WGS, ChIP-seq, ATAC-seq, or amplicon pipelines
• Running bioinformatics workflows on SLURM/PBS/SGE clusters
• Scaling pipelines to AWS, GCP, or Azure
• Containerizing workflows with Docker or Singularity
• Debugging failed pipeline steps and log inspection
• Managing conda environments within workflows
• Parallelizing sample processing and step execution
• Generating reproducible workflow reports

Quick Start

Snakemake RNA-seq Pipeline

# Snakefile for bulk RNA-seq analysis
SAMPLES = ["sample1", "sample2", "sample3", "sample4"]
GENOME = "GRCh38"

rule all:
    input:
        expand("results/counts/{sample}.counts.txt", sample=SAMPLES),
        "results/multiqc_report.html",

rule trim_reads:
    input:
        r1="data/raw/{sample}_R1.fastq.gz",
        r2="data/raw/{sample}_R2.fastq.gz",
    output:
        r1="data/trimmed/{sample}_R1_trimmed.fastq.gz",
        r2="data/trimmed/{sample}_R2_trimmed.fastq.gz",
        json="qc/{sample}_fastp.json",
        html="qc/{sample}_fastp.html",
    threads: 8
    shell:
        """
        fastp -i {input.r1} -I {input.r2} \
              -o {output.r1} -O {output.r2} \
              -j {output.json} -h {output.html} \
              --thread {threads} --detect_adapter_for_pe
        """

rule align_star:
    input:
        r1="data/trimmed/{sample}_R1_trimmed.fastq.gz",
        r2="data/trimmed/{sample}_R2_trimmed.fastq.gz",
        index="reference/star_index/",
    output:
        bam="results/bam/{sample}.Aligned.sortedByCoord.out.bam",
        log="results/bam/{sample}.Log.final.out",
    threads: 16
    shell:
        """
        STAR --runThreadN {threads} \
             --genomeDir {input.index} \
             --readFilesIn {input.r1} {input.r2} \
             --readFilesCommand zcat \
             --outSAMtype BAM SortedByCoordinate \
             --outSAMattributes NH HI AS NM \
             --outFileNamePrefix results/bam/{wildcards.sample}. \
             --quantMode GeneCounts
        samtools index {output.bam}
        """

rule feature_counts:
    input:
        bam="results/bam/{sample}.Aligned.sortedByCoord.out.bam",
        gtf="reference/annotation.gtf",
    output:
        counts="results/counts/{sample}.counts.txt",
    threads: 4
    shell:
        """
        featureCounts -T {threads} \
                      -a {input.gtf} \
                      -o {output.counts} \
                      -p -B -C \
                      {input.bam}
        """

rule multiqc:
    input:
        expand("qc/{sample}_fastp.json", sample=SAMPLES),
        expand("results/bam/{sample}.Log.final.out", sample=SAMPLES),
        expand("results/counts/{sample}.counts.txt", sample=SAMPLES),
    output:
        "results/multiqc_report.html",
    shell:
        "multiqc qc/ results/ -o results/ -n multiqc_report"