heartmula

HeartMuLa - Open-Source Music Generation

Overview

HeartMuLa is a family of open-source music foundation models (Apache-2.0) that generates music conditioned on lyrics and tags. Comparable to Suno for open-source. Includes:

• HeartMuLa - Music language model (3B/7B) for generation from lyrics + tags
• HeartCodec - 12.5Hz music codec for high-fidelity audio reconstruction
• HeartTranscriptor - Whisper-based lyrics transcription
• HeartCLAP - Audio-text alignment model

When to Use

• User wants to generate music/songs from text descriptions
• User wants an open-source Suno alternative
• User wants local/offline music generation
• User asks about HeartMuLa, heartlib, or AI music generation

Hardware Requirements

• Minimum: 8GB VRAM with --lazy_load true (loads/unloads models sequentially)
• Recommended: 16GB+ VRAM for comfortable single-GPU usage
• Multi-GPU: Use --mula_device cuda:0 --codec_device cuda:1 to split across GPUs
• 3B model with lazy_load peaks at ~6.2GB VRAM

Installation Steps

1. Clone Repository

cd ~/  # or desired directory
git clone https://github.com/HeartMuLa/heartlib.git
cd heartlib

2. Create Virtual Environment (Python 3.10 required)

uv venv --python 3.10 .venv
. .venv/bin/activate
uv pip install -e .

3. Fix Dependency Compatibility Issues

IMPORTANT: As of Feb 2026, the pinned dependencies have conflicts with newer packages. Apply these fixes:

# Upgrade datasets (old version incompatible with current pyarrow)
uv pip install --upgrade datasets

# Upgrade transformers (needed for huggingface-hub 1.x compatibility)
uv pip install --upgrade transformers

4. Patch Source Code (Required for transformers 5.x)

Patch 1 - RoPE cache fix in src/heartlib/heartmula/modeling_heartmula.py:

In the setup_caches method of the HeartMuLa class, add RoPE reinitialization after the reset_caches try/except block and before the with device: block:

# Re-initialize RoPE caches that were skipped during meta-device loading
from torchtune.models.llama3_1._position_embeddings import Llama3ScaledRoPE
for module in self.modules():
    if isinstance(module, Llama3ScaledRoPE) and not module.is_cache_built:
        module.rope_init()
        module.to(device)