SeqLens v2 Micro Novel 16K

A compact genomic language model pre-trained with a novel recipe — span masking + complexity-weighted MLM loss — on prokaryotic genomes. Achieves 96.2% on coding/non-coding classification with only 10M parameters.

What's Different (Novel Pre-training Recipe)

This model uses the same BiMamba2 architecture as seqlens-v2-micro-16k but a different pre-training recipe:

Why this matters: Random masking teaches the model to predict individual nucleotides from local context — a shallow task that saturates quickly. Span masking forces prediction over contiguous regions (codons, promoters, gene boundaries), learning higher-order structural patterns. Complexity weighting prevents the model from wasting capacity on repetitive DNA (~30% of prokaryotic genomes), focusing learning on biologically informative regions.

Results

Coding vs Non-coding (linear probe, frozen backbone)

Genus Classification (50 held-out genera, linear probe, frozen backbone)

Embedding Clustering

Extended Benchmark Comparison

All evaluations below use the frozen linear probe protocol (frozen embeddings → LogisticRegression). Published fine-tuned baselines (e.g., ProkBERT's fine-tuned MCC scores) are not directly comparable — this is an apples-to-apples comparison across models with a fixed downstream classifier.

ProkBERT Prokaryotic Benchmarks (accuracy)

NTv3-8M leads this suite (trained on 9T bp across all species, ~36× more data than our 248B prokaryotic tokens). Interestingly, v2_standard edges out this model (v2_novel) on the ProkBERT suite — the opposite ranking from CDS classification and GenomicBenchmarks, where the novel recipe wins. See PROJECT_STATE.md §3.4 for a per-task recipe breakdown.

GenomicBenchmarks — Eukaryotic Tasks (accuracy)

The human_ensembl_regulatory task is excluded: our single-nucleotide tokenizer preserves exact sequence length, and length alone is a near-complete shortcut for this task's 3-way label (a Random Forest trained on sequence length alone reaches 91.3% accuracy). Our models score ~0.99–1.00 on this task as an artifact of that leakage, not because of learned regulatory biology. Despite training exclusively on prokaryotic genomes, our 10M models remain competitive with human/multi-species models up to 10× our size on the remaining 7 eukaryotic tasks.

Full results: experiments/eval_reports/genomic_benchmarks/comparison.json and experiments/eval_reports/prokbench/comparison.json.

Architecture

Training

Pre-training Recipe

Optimizer & Schedule

Data & Compute

Usage

import torch
from model import SeqLensForMLM
from config import MICRO_CONFIG
from tokenizer import NucleotideTokenizer

device = torch.device("cuda")
ckpt = torch.load("seqlens-v2-micro-novel-16k.pt", map_location=device)
model = SeqLensForMLM(MICRO_CONFIG).to(device).to(torch.bfloat16)
model.load_state_dict(ckpt["model"])
model.eval()

tokenizer = NucleotideTokenizer(max_len=16384)
seq = "ATGCGATCG..."
token_ids = torch.tensor([tokenizer.encode(seq)], dtype=torch.long).to(device)

# Sequence-level embedding (for classification, retrieval)
with torch.no_grad():
    emb = model.get_embeddings(token_ids, pool="attention")  # (1, 256)

# Per-position logits (for MLM, motif detection)
with torch.no_grad():
    output = model(token_ids)
    logits = output["logits"]  # (1, L, 9)

Model Family

Citation

@misc{seqlens-v2-novel-2026,
  title={SeqLens v2: Span-Masked Complexity-Weighted Pre-training for Compact Genomic Language Models},
  author={SeqSight Team},
  year={2026},
  url={https://huggingface.co/seqSight/seqlens-v2-micro-novel-16k}
}

License

Apache 2.0