Blog | Carbon-X | Carbon-X-Base | Hugging Face Bio | Code

Carbon-X-SFT

The supervised fine-tuned model behind Carbon-X: Carbon-X-Base after fine-tuning on DNA, biology and instruction-following conversations, before reinforcement learning.

Table of Contents

1. Model Summary
2. How to use
3. Evaluation
4. Training
5. Limitations
6. License
7. Acknowledgements

Model Summary

Carbon-X-SFT is a 14.8B-parameter decoder-only language model for DNA understanding, trained jointly on DNA sequences and natural-language text. You give it raw DNA inside an ordinary chat conversation and it answers in English: it predicts whether mutations are benign or pathogenic, identifies regulatory regions, classifies sequences by organism group, predicts the next bases of a sequence, and explains its reasoning.

The Carbon-X models are trained in three stages:

1. Carbon-X-Base: Qwen3-14B with a DNA vocabulary, mid-trained on 302B tokens of DNA and text.
2. Carbon-X-SFT (this model): Carbon-X-Base fine-tuned for 3 epochs on a mixture of 20 DNA, biology and instruction datasets.
3. Carbon-X: Carbon-X-SFT trained further with reinforcement learning and verifiable rewards on DNA tasks.

Carbon-X-SFT averages 59.3% across our five DNA benchmarks with thinking disabled, against 40.0% for Qwen3-14B and 66.7% for Carbon-X (Evaluation). For the best results, use Carbon-X. Carbon-X-SFT is useful as a starting point for your own reinforcement learning, or as a reference for what supervised fine-tuning alone achieves.

Key features

• 14.8B parameters: 40 layers, hidden size 5120, GQA (40 query heads, 8 KV heads), SwiGLU, RMSNorm with QK-norm.
• Hybrid tokenizer: the Qwen3 BPE vocabulary for text plus 5,515 DNA tokens. These are 4,096 6-mers, 1,364 tail tokens for sequences whose length is not a multiple of 6, and the <dna>, </dna> and <oov> markers. DNA in a prompt is detected and tagged automatically.
• Chat model: fine-tuned and evaluated with thinking disabled.
• Context window: 40,960 tokens.
• Efficient inference: serves with vLLM on a single GPU.

How to use

Carbon-X-SFT uses a standard causal LM architecture. The DNA tokenizer is custom code, so pass trust_remote_code=True when you load the tokenizer. Carbon-X-SFT requires Transformers v5 or later.

pip install -U transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "HuggingFaceBio/Carbon-X-SFT"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda")

messages = [
    {
        "role": "user",
        "content": (
            "Reference: ATGGTGCATCTGACTCCTGAGGAGAAGTCTGCCGTTACTGCCCTGTGGGGCAAGG\n"
            "Variant:   ATGGTGCATCTGACTCCTGTGGAGAAGTCTGCCGTTACTGCCCTGTGGGGCAAGG\n\n"
            "Is this variant likely to be benign or pathogenic? Explain your reasoning."
        ),
    }
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=1024)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Carbon-X-SFT was fine-tuned and evaluated with enable_thinking=False, and we recommend keeping it that way. generate() uses the sampling defaults from generation_config.json (temperature 0.6, top-p 0.95, top-k 20), which are also the settings we used for every reported result.

Serving with vLLM

pip install -U vllm
vllm serve HuggingFaceBio/Carbon-X-SFT --trust-remote-code

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="HuggingFaceBio/Carbon-X-SFT",
    messages=[{"role": "user", "content": "What does this sequence encode?\n\nATGGTGCATCTGACTCCTGAGGAGAAGTCTGCCGTT"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)

vLLM also applies the sampling defaults from generation_config.json unless you override them in the request.

Tokenizer: working with DNA inputs

The tokenizer switches between BPE for text and 6-mers for DNA. You can paste sequences directly into a message:

>>> ids = tokenizer("Describe this sequence: ACGTACGATTCA")["input_ids"]
>>> tokenizer.convert_ids_to_tokens(ids)
['Describe', 'Ġthis', 'Ġsequence', ':', 'Ġ', '<dna>', 'ACGTAC', 'GATTCA', '</dna>']

• DNA is tagged automatically. Runs of at least 6 A/C/G/T bases are wrapped in <dna>…</dna> and tokenized as 6-mers. Runs of at least 12 bases with up to 20% IUPAC ambiguity codes (N, R, Y, …) are tagged too. Lowercase bases are uppercased, and sequences displayed in space-separated blocks are joined. Text you have already wrapped in <dna>…</dna> is always tokenized as DNA. To turn off detection for a call, pass auto_dna_tags=False to the tokenizer or to apply_chat_template().
• Only A, C, G and T form 6-mers. A 6-mer that contains any other character, such as N, becomes a single <oov> token. To keep full resolution, filter to canonical bases first.
• Any length works. When a sequence's length is not a multiple of 6, its last 1–5 bases are encoded with dedicated tail tokens, and decoding returns the original sequence exactly.
• DNA in responses is tagged. The model writes DNA inside <dna>…</dna>. Decoding with skip_special_tokens=True removes the tags.
• First load needs Hub access. On first load, the tokenizer downloads the pinned BPE vocabulary it builds on.

Prompting tips

• Put the sequence and the question in the same user message.
• Include any context you already have, such as the gene, strand, coordinates or annotations. Without it, the model may guess at a sequence's identity or function.
• For variant questions, give the reference and alternate sequences, or a reference sequence with the position and the base change.

Evaluation

We evaluate on the same five benchmarks as Carbon-X, with thinking disabled, the default sampling settings and one sample per problem. The table shows how much of Carbon-X's performance comes from supervised fine-tuning and how much reinforcement learning adds:

Qwen3-14B, the model both start from, averages 40.0%.

• BioReason Coding (1,018 variants, balanced accuracy): given original and mutated sequences from protein-coding regions, predict whether the mutation is benign or pathogenic. The gene name is removed from the prompt, so the model must work from the sequence alone.
• BioReason Non-SNV (873 variants, balanced accuracy): predict whether insertions, deletions and other changes that are not single-nucleotide variants are benign or pathogenic.
• GenomeQA (2,600 questions, accuracy averaged over six tasks): multiple-choice questions about sequence features: enhancer vs. promoter, splice sites, organism group, histone marks, transcription-factor binding sites and motifs.
• NextBase (600 sequences, per-base accuracy): predict the next DNA bases of a sequence. The score is the fraction of bases predicted correctly.
• FrontierVEP (1,000 variants, accuracy): given a long reference sequence (up to 8,192 bases) and a single-nucleotide variant, predict whether the variant is benign or pathogenic (500 benign, 500 pathogenic).

GenomeQA per-task accuracy

Carbon-X-SFT also remains a capable assistant. On IFEval it reaches 75.6% prompt-level strict accuracy, or 79.9% averaged over the four IFEval metrics.

All training data was decontaminated against these benchmarks with three checks: whole-span DNA overlap, text n-gram overlap and sequence homology search.

Training

Model

• Architecture: decoder-only Transformer with 40 layers, hidden size 5120, FFN size 17408, 40 attention heads with GQA (8 KV heads), head dimension 128, SwiGLU, RMSNorm with QK-norm and untied embeddings.
• Positional embedding: RoPE with base θ = 1 × 10^6. The context window is 40,960 tokens.
• Vocabulary: 157,184 tokens: 151,669 BPE tokens and 5,515 DNA tokens, including padding to a multiple of 128.
• Precision: bfloat16.

Mid-training

Carbon-X-SFT starts from Carbon-X-Base: Qwen3-14B with DNA tokens added to its vocabulary, trained for 302B tokens on a mixture of 55% raw DNA, 15% biology text, 20% English and 10% instruction data. See the Carbon-X-Base card for the full data mixture and schedule.

Supervised fine-tuning

Carbon-X-Base is fine-tuned for 3 epochs on a mixture of 20 dataset subsets, about 204K conversations per epoch. The learning rate follows a cosine schedule from 1e-5 to 1e-6 after a 3% warmup, with a global batch size of 32, a maximum gradient norm of 0.2, examples of up to 9,984 tokens and the loss computed on assistant tokens only. The mixture covers:

• Variant-effect reasoning over several views of the same variant (about 30% of the mixture)
• Sequence classification questions: enhancers vs. promoters, organism groups, transcription-factor motifs and binding sites
• Basic DNA operations, plus procedurally generated DNA skills such as counting, searching and editing sequences (about 10%)
• Sequence recovery and next-base prediction
• Biology questions from community Q&A forums
• Instruction-following, format-following and general chat data, which keep the model a good assistant

Software

• Fine-tuning: TRL SFTTrainer.

Limitations

Carbon-X-SFT is a research checkpoint. It has not been clinically validated, and it should not be used to make medical decisions.

• ⚠️ Genetic data is highly sensitive. Depending on how you use this model (local download, inference API or endpoints, third-party inference providers, Spaces demos or others), input and output data may be processed or handled differently by different providers or Space owners. Make sure you understand and agree with how your data is handled before using the model.
• Weaker than Carbon-X. Reinforcement learning adds 7.4 points on average, and 19.9 points on BioReason Non-SNV. Use Carbon-X unless you need the fine-tuned model specifically.
• Exact sequence operations. The model can miscount bases, confuse coordinates or make incorrect edits, particularly when it locates exact positions in long sequences. For tasks that need exact operations, pair it with tools that perform them.
• Missing context and hallucinations. The model may invent a sequence's identity or function when the prompt does not give enough information. Include known annotations and context where available, and verify its biological claims.
• Uneven task coverage. Splice-site and histone-mark questions remain weak (29.0% and 28.6% on GenomeQA).
• Long sequence generation. Quality degrades when the model is asked to generate long DNA sequences. It reads and explains existing sequences. It was not trained to design new ones.
• Long inputs. Mid-training used 8,192-token sequences (up to ≈ 49 kbp of DNA), and fine-tuning examples were at most 9,984 tokens. The model accepts up to 40,960 tokens, but we have not evaluated inputs much longer than those seen in training.
• Thinking mode. The chat template supports enable_thinking=True, but Carbon-X-SFT was fine-tuned and evaluated with thinking disabled.
• No GGUF support. llama.cpp and other GGUF runtimes cannot load the custom Python DNA tokenizer, so DNA would be tokenized incorrectly.

License

Apache 2.0.

Acknowledgements

Carbon-X is a joint collaboration between the research teams at Hugging Face, Zhongguancun Academy, and the Telethon Institute of Genetics and Medicine/University of Naples “Federico II”.