rnafm

RNA-FM: Foundation Model for Non-Coding RNA

Overview

RNA-FM is a BERT-style transformer pretrained on 23 million non-coding RNA (ncRNA) sequences via masked language modeling. It generates contextual nucleotide embeddings that capture RNA structure and function without labeled data.

Model variants:

• RNA-FM — trained on ncRNA sequences (23M); embedding dim = 640; tokenizes individual nucleotides
• mRNA-FM — trained on 45M mRNA coding sequences (CDS); embedding dim = 1280; tokenizes 3-mers (codons)

Capabilities:

• Sequence embeddings — per-token or pooled representations for any downstream task
• Secondary structure prediction — base-pair contact maps (outperforms LinearFold/SPOT-RNA)
• RNA family clustering — zero-shot separation of RNA families in embedding space
• Functional prediction — UTR function, gene expression, RNA-protein binding

API note: RNA-FM's Python API mirrors ESM2 — (model, alphabet) pair, batch_converter, repr_layers. If you know ESM2, RNA-FM will feel familiar.

Installation

pip install rna-fm

Requirements: Python ≥ 3.8, PyTorch ≥ 1.9. GPU with CUDA 11.1+ recommended.

Model weights download automatically on first use (~1.2 GB for RNA-FM, ~957 MB for mRNA-FM).

Model Checkpoints

Checkpoint	Training data	Embed dim	Tokenization	Best for
rna_fm_t12	23M ncRNA sequences	640	Single nucleotide	ncRNA, structural RNA, general RNA
mrna_fm_t12	45M mRNA CDS sequences	1280	3-mer (codon)	mRNA analysis, codon usage, translation

Core Usage

Load model

import fm

# ncRNA model (default)
model, alphabet = fm.pretrained.rna_fm_t12()

# mRNA model
model, alphabet = fm.pretrained.mrna_fm_t12()

model.eval()

Extract embeddings

import torch
import fm

model, alphabet = fm.pretrained.rna_fm_t12()
model.eval()

batch_converter = alphabet.get_batch_converter()

# Input: list of (label, sequence) tuples
sequences = [
    ("rna1", "GGGUGCGAUCAUACCAGCACUAAUGCCCUCCUGGGAAGUCCUCGUGUUGCACCCCU"),
    ("rna2", "AUGUAAGGCCUUGUAACGCUCUAAACUUCCCCCGCGACGUUUUU"),
]

batch_labels, batch_strs, batch_tokens = batch_converter(sequences)

with torch.no_grad():
    results = model(batch_tokens, repr_layers=[12])

# Per-token embeddings from last layer: (batch, seq_len, 640)
token_embeddings = results["representations"][12]

# Per-sequence mean pooling (exclude BOS/EOS/PAD)
padding_idx = alphabet.padding_idx
for i, label in enumerate(batch_labels):
    mask = (batch_tokens[i] != padding_idx).float()
    seq_emb = (token_embeddings[i] * mask.unsqueeze(-1)).sum(0) / mask.sum()
    print(f"{label}: {seq_emb.shape}")  # (640,)