alterlab-tiledbvcf

TileDB-VCF

Overview

TileDB-VCF is a high-performance C++ library with Python and CLI interfaces for efficient storage and retrieval of genomic variant-call data. Built on TileDB's sparse array technology, it enables scalable ingestion of VCF/BCF files, incremental sample addition without expensive merging operations, and efficient parallel queries of variant data stored locally or in the cloud.

When to Use This Skill

This skill should be used when:

• Building a queryable, compressed variant store from many single-sample VCF/BCF files (cohort/population datasets too large for flat VCF)
• Incrementally adding new samples to an existing store without re-merging
• Querying specific genomic regions across many samples (region/sample-partitioned reads)
• Exporting region/sample subsets back to VCF/BCF for downstream tools
• Working with variant data on cloud storage (S3, Azure, GCS) or TileDB Cloud
• Prototyping or teaching scalable genomics-variant workflows

Quick Start

Installation

Preferred method: conda/mamba from the tiledb channel. tiledbvcf-py is NOT on PyPI, conda-forge, or bioconda — it ships from the tiledb Anaconda channel, with native osx-arm64 builds (no Rosetta/CONDA_SUBDIR workaround needed on Apple Silicon). Supports Python 3.9–3.12.

# Native Apple Silicon (osx-arm64) — also works on osx-64 / linux-64
conda create -n tiledb-vcf -c conda-forge -c tiledb \
  python=3.12 tiledbvcf-py=0.40 pandas pyarrow numpy
conda activate tiledb-vcf

Alternative: Docker images (pulls the CLI/Python interface; latest tag tracks current release)

docker pull tiledb/tiledbvcf-py     # Python interface
docker pull tiledb/tiledbvcf-cli    # Command-line interface

Basic Examples

Create and populate a dataset:

import tiledbvcf

# Create a new dataset
ds = tiledbvcf.Dataset(uri="my_dataset", mode="w",
                      cfg=tiledbvcf.ReadConfig(memory_budget_mb=1024))

# Ingest VCF files (must be single-sample with indexes)
# Requirements:
# - VCFs must be single-sample (not multi-sample)
# - Must have indexes: .csi (bcftools) or .tbi (tabix)
ds.ingest_samples(["sample1.vcf.gz", "sample2.vcf.gz"])