加载中…
加载中…
CENO-600M-A230M-1m is the long-context (1M) checkpoint of the 600M CENO DNA foundation model — a causal language model over genomic sequence built on a Nemotron-H Mamba / Attention / Mixture-of-Experts hybrid backbone (no MSA inputs).
It is part of the CENO DNA foundation model family. Model code, the VEP pipeline, and a
generation demo live in the companion CENO code repository.
This checkpoint is standalone-loadable with trust_remote_code=True — the model code is
bundled here.
| Family | CENO (base) |
| Training stage | Long-context extension (stage 4) |
| Training context length | 1,048,576 tokens |
| Parameters | 630.8M |
| Precision | bfloat16 |
model_type |
ceno |
| Architecture class | CENOForCausalLM |
| Auto-map (model) | modeling_ceno.CENOForCausalLM |
| Auto-map (tokenizer) | ceno_tokenizer.CENOCharLevelTokenizer |
CENO's Mamba / Attention / MoE backbone has no fixed context window. The training context length above is the sequence length this checkpoint was trained at — not a hard limit.
max_position_embeddingsinconfig.jsonis nominal and non-restricting.
| Property | Value |
|---|---|
| Hidden layers | 20 |
| Hidden size | 1024 |
| Attention heads | 16 |
| Intermediate size | 4096 |
| Experts (MoE) | 8 (top-2 per token) |
| Vocabulary | 512 (byte / character-level) |
The backbone is a Mamba / Attention / Mixture-of-Experts hybrid (Nemotron-H architecture). The tokenizer is character-level, mapping DNA bases to their ASCII byte codes.
from transformers import AutoModelForCausalLM, AutoTokenizer
ckpt = "CladeTeam/CENO-600M-A230M-1m"
model = AutoModelForCausalLM.from_pretrained(ckpt, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(ckpt, trust_remote_code=True)
ids = tokenizer.encode("ATCGATCG", return_tensors="pt")
# out = model.generate(ids, max_new_tokens=128) # needs a CUDA GPU (Mamba kernels)
The Mamba layers require CUDA kernels, so forward passes and generation need a GPU. Config, tokenizer, and weight loading are CPU-safe.
CENO-*) — genomic-sequence generation and embedding extraction;
downstream adaptation (fine-tuning, probing) for genomics tasks.CENO-P-*) — variant effect prediction (VEP) by scoring wild-type
vs. variant sequences with delta log-likelihood. See the TraitGym VEP example in the
CENO code repository.Apache-2.0. The bundled model code is derived from NVIDIA's Nemotron-H Hugging Face
implementation (Apache-2.0); the tokenizer is derived from the Arc Institute Evo2
CharLevelTokenizer (Apache-2.0). See the LICENSE and NOTICE files in this repository
for full attribution.
Current source and usage instructions are maintained at CladeTeam/CENO.
Loading
uses the checkpoint dtype by default; explicit caller dtype overrides are honored.
Use torch_dtype="auto", .cuda().eval() and torch.inference_mode() for raw
forward calls. generate() enters no-grad mode automatically.
The HF precision path is single-GPU, unpadded, and permits at most 131072 tokens per prefill. Use the repository's vLLM entry for larger prefills.
The tested HF stack is PyTorch 2.7.0a0 (NVIDIA 25.03), Transformers 4.48.3, mamba-ssm 2.2.4, causal-conv1d 1.2.2.post1 and FlashAttention 2.7.3. HF and vLLM use separate environments. Numerical correctness is not a guarantee of repetition-free or biologically functional generation.