Canonical Mamba-3 DNA language model (trained in mixlab)

A small canonical Mamba-3 DNA language model, pretrained on human genome in mixlab on a single Apple-silicon Mac. It is a reusable backbone: load it, then fine-tune a classifier on your own genomic task. On the human_enhancers_cohn benchmark, a classifier fine-tuned on this backbone reaches 0.728 accuracy / 0.811 AUROC, ahead of a same-size attention model (0.717) and the benchmark's baseline CNN (~0.69-0.70).

This is a model to learn from and build on, not a state-of-the-art result. The point is that a real selective-scan state-space model (Mamba-3), trained on a laptop on only three chromosomes, is competitive with much larger whole-genome-pretrained models, and that swapping the mixer to get there is a config edit in mixlab, not new model code.

What it is

• 14.34M parameters: 6 mamba3-canonical blocks interleaved with 6 SwiGLU blocks, model_dim 384, seq_len 512, nucleotide vocab 9 (A/C/G/T/N + BOS/EOS/PAD/MASK). Mamba-3 = selective-scan SSM with a complex-valued state update, a multi-input multi-output (MIMO) B/C formulation, and the exponential-trapezoidal recurrence (Lahoti et al. 2026).
• Pretrained next-base on GRCh38 chromosomes 1-3 (a continuous nucleotide stream, ~562M tokens), 20,000 steps, with reverse-complement augmentation. Validation loss 1.13.

Load it

This is a custom-code export, so load with trust_remote_code=True. The token ids are BOS=1, EOS=2, PAD=0, and A/C/G/T/N = 4/5/6/7/8.

import torch
from transformers import AutoModel

repo = "mrothroc/dna-enhancers-mamba3-mixlab"
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()   # backbone hidden states

ids = torch.tensor([[1, 4, 5, 6, 7, 4, 5, 2]])   # BOS A C G T A C EOS
hidden = model(input_ids=ids).last_hidden_state   # [1, 8, 384]

To reproduce the enhancer classifier, fine-tune with AutoModelForSequenceClassification (or use mixlab's native classification training). The full recipe is in the cookbook below.

Reproduce / learn more

Full recipe (reproduce an attention baseline, then improve it by swapping the mixer to Mamba, all on a laptop): mixlab cookbook, genomics-mamba. The same-size attention baseline is at mrothroc/dna-enhancers-attention-mixlab.

For the story behind it, read I Built an ML Architecture Lab in Go.

Citation

Please cite what this builds on:

• Mamba-3 (the architecture of the mixer): Lahoti, A.; Li, K. Y.; Chen, B.; Wang, C.; Bick, A.; Kolter, J. Z.; Dao, T.; Gu, A. "Mamba-3: Improved Sequence Modeling using State Space Principles." ICLR 2026. arXiv:2603.15569
• Mamba (foundational selective SSM): Gu, A.; Dao, T. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." 2023. arXiv:2312.00752
• Genomic Benchmarks (the task): Gresova, K. et al. "Genomic benchmarks: a collection of datasets for genomic sequence classification." BMC Genomic Data 2023, 24, 25. doi:10.1186/s12863-023-01123-8
• GRCh38 human reference genome (Genome Reference Consortium), via the UCSC Genome Browser: Kent, W. J. et al. "The Human Genome Browser at UCSC." Genome Research 2002, 12 (6), 996-1006. doi:10.1101/gr.229102