genetic-dna-tiny

A small DNA language model (6.3M parameters) trained from scratch on the human reference genome GRCh37/hg19, built to learn how genomic language models work.

Not for clinical use. Scores from this model are not medical advice or a diagnosis.

Model

• Llama-style decoder (LlamaForCausalLM): hidden 256, 6 layers, 8 heads, intermediate 1024
• Context: 1024 bases (+ [BOS]/[EOS])
• Objective: next-base prediction (causal language modeling)
• Tokenizer: character level, [PAD] [UNK] [BOS] [EOS] A C G T N. One base = one token, so a single-base change alters exactly one token. Lowercase (soft-masked) input is uppercased and any other letter becomes N.

Training data

• UCSC hg19 chromosomes from hgdownload.soe.ucsc.edu
• Train: chr22 (both strands), validation: chr21 (+ strand)
• Non-overlapping 1024 bp windows, dropping windows with more than 1% unknown bases (N)
• 4000 steps, batch size 16, learning rate 0.001 (cosine)

Evaluation

Bits per base on chr21 (35,087,360 bases in windows without N), each base predicted from the bases before it in its window. Uniform guessing = 2.0; lower is better.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("rubenich/genetic-dna-tiny")
model = AutoModelForCausalLM.from_pretrained("rubenich/genetic-dna-tiny")

ids = tok("ACGTTGCA", return_tensors="pt")["input_ids"]
with torch.no_grad():
    logits = model(input_ids=ids).logits

Zero-shot variant scoring compares log P(alt window) - log P(ref window) on an hg19 window around a single-base change; see genetic_model.score_variant in the source repo.

Limitations

• Trained on 1 chromosome(s) of one reference genome: it has not seen human variation, other chromosomes, or other species.
• Tiny model and short context: it captures local sequence statistics, not long-range regulatory structure.
• Coordinates and alleles must be GRCh37/hg19, + strand.