large-scale-single-cell-matrix-loading

large-scale-single-cell-matrix-loading

Summary

Load and initialize single-cell count matrices with ≥10 million cells in compressed sparse row (CSR) format for scalable downstream analysis. This skill is essential when working with very large omics datasets where memory efficiency and linear-time complexity are critical.

When to use

Apply this skill when you have a single-cell count matrix with 10 million or more cells that must be processed through dimension reduction, clustering, or integration pipelines. Use it specifically before executing matrix-free spectral embedding (tl.spectral) or other scalable SnapATAC2 tools that require efficient sparse matrix representation.

When NOT to use

• Input is already a processed and filtered AnnData object ready for dimension reduction — use directly without re-initialization.
• Dataset contains fewer than 100,000 cells — standard dense matrix handling may be more efficient.
• Matrix is already in a non-CSR sparse format (e.g., COO, LIL) incompatible with SnapATAC2 backend — convert format explicitly first.

Inputs

• single-cell count matrix with ≥10 million cells (dense or sparse format)
• cell-by-feature table (BAM files, fragment files, or pre-computed count matrices)
• metadata specifying matrix dimensions and cell/feature identifiers

Outputs

• CSR-formatted sparse matrix compatible with SnapATAC2 backend
• AnnData object (.h5ad) with initialized count matrix
• memory usage profile and initialization metrics

How to apply

Load the count matrix using SnapATAC2's dataset utilities or external data sources and initialize it in compressed sparse row (CSR) format, which is the sparse matrix representation required by SnapATAC2's backend. Verify that the matrix dimensions match the expected number of cells (rows) and features (columns), and confirm that the data type supports the downstream cosine similarity metric used in spectral decomposition. Monitor memory footprint during initialization using a profiler such as psutil or memory_profiler to ensure the sparse format achieves the expected linear space complexity. Document the initialization parameters and resulting matrix structure (sparsity percentage, number of nonzero elements) before proceeding to decomposition or embedding steps.