Coli-ByT5-Small

Byte-level Sequence-to-Sequence Model for Synonymous Codon Optimization in Escherichia coli

Model Description

• Developed by: Aakash Dandekar
• Model Type: Byte-level Encoder-Decoder Transformer (T5ForConditionalGeneration / ByT5)
• Base Model: google/byt5-small
• Precision: float16
• Primary Task: Protein-to-DNA synonymous codon translation and optimization for Escherichia coli expression

Overview & Biological Motivation

Coli-ByT5-Small is a specialized byte-level sequence-to-sequence language model fine-tuned to solve the synonymous codon optimization task for the model bacterium Escherichia coli.

Given an amino acid sequence, the model generates an optimized synonymous nucleotide coding sequence (CDS). Unlike classical heuristic tools that rely on naive frequency maximization tables (which often eliminate natural translational pauses, disrupt co-translational folding, or destroy the 5' translational ramp), Coli-Byt5-Small learns contextual codon harmonization from validated endogenous sequences.

Why Byte-Level (ByT5)?

By operating directly on UTF-8 bytes rather than subword tokens:

• No Tokenizer Fragmentation: Biological sequences (amino acid letters ACDEFGHIKLMNPQRSTVWY and nucleotides ATGC) are represented at single-character granularity without out-of-vocabulary (OOV) tokens.
• Strict In-Frame Modeling: Enables fine-grained contextual awareness across codon triplets and dicodon boundaries without subword boundary mismatch.

---

Quickstart & Inference

Installation

pip install torch transformers sentencepiece

Direct Generation with Transformers

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "aakashdandekar2006/Coli-Byt5-Small"  # or local directory path

# Load byte-level tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Input amino acid sequence (e.g., GFP fragment / peptide sequence)
protein_seq = "MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEG"
prompt = f"Optimize codon sequence for Escherichia coli: {protein_seq}"

# Tokenize byte input
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Autoregressive generation
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_length=len(protein_seq) * 3 + 64,
        num_beams=4,
        early_stopping=True
    )

optimized_dna = tokenizer.decode(outputs[0], skip_special_tokens=True)
print("Input Protein:", protein_seq)
print("Optimized DNA:", optimized_dna)

Hugging Face Pipeline Usage

from transformers import pipeline

pipe = pipeline(
    "text2text-generation",
    model="aakashdandekar2006/Coli-ByT5-Small",
    torch_dtype="float16",
    device_map="auto"
)

result = pipe("Optimize codon sequence for Escherichia coli: MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEG")
print(result[0]["generated_text"])

---

Training Details

• Dataset: Endogenous and curated Escherichia coli coding sequence pairs (CDS) sourced from adibvafa/CodonTransformer and NCBI reference genomes. Filtered through biological schema verification:
  • 100% in-frame translation sequence identity (zero missense or nonsense substitutions)
  • Strict IUPAC amino acid compliance (ACDEFGHIKLMNPQRSTVWY) and canonical base compliance (A, C, G, T)
  • Exact nucleotide-to-protein length correspondence ($\text{length}(\text{DNA}) = 3 \times \text{length}(\text{Protein})$)
  • Strict absence of premature internal stop codons
• Architecture: 12 encoder blocks, 4 decoder blocks, 6 attention heads, hidden dimension $d_{\text{model}} = 1472$, feed-forward dimension $d_{\text{ff}} = 3584$ (~300.77M total parameters).
• Adaptation Methodology: Parameter-Efficient Fine-Tuning via LoRA (r=16, lora_alpha=32, target_modules="all-linear") using peft and Seq2SeqTrainer, subsequently merged into standalone safetensors float16 weights.
• Hardware: 1x NVIDIA RTX GPU (CUDA-accelerated)
• Epochs / Steps: 3 Epochs
• Learning Rate: 5e-4 with cosine learning rate scheduler and AdamW optimizer
• Mixed Precision: FP16

Biological Verification & Evaluation Standards

---

Benchmark Results

Comparative evaluation of Coli-ByT5-Small against a traditional frequency-maximizing greedy baseline on a test sequence of 539 aa (1617 nt expected):

Sequence translated successfully with zero amino acid mismatches.

Sequence Comparison (First 60 Nucleotides)

• Model:

  ATGGAAGATGCAAAAAATATTAAAAAAGGTCCTGCACCATTTTATCCGCTGGAAGATGGC...
  

• Naive Greedy:

  ATGGAAGATGCGAAAAACATTAAAAAAGGCCCGGCGCCGTTTTATCCGCTGGAAGATGGC...
  

Analysis & Insights

• Translation Fidelity: The model achieved 100% translation accuracy across all 539 amino acids with zero missense or premature termination mutations.
• Balanced Codon Adaptation: Traditional greedy substitution achieves an artificial CAI of 1.000, which can exhaust specific tRNA pools and disrupt natural translational pause sites. Coli-ByT5-Small achieves a harmonized CAI of 0.884, matching natural high-expression transcripts.
• Synthesis Safety: Coli-ByT5-Small produced 0 forbidden restriction enzyme recognition sites, avoiding the cloning and synthesis motif conflict introduced by the naive algorithm (1 site).

---

Limitations & Ethical Considerations

• Host Specificity: This model is tuned specifically for Escherichia coli. Codon bias, tRNA pools, and translation kinetics differ substantially in other hosts (e.g., yeast, human, plant, insect expression systems).
• Secondary Structure & Experimental Verification: While the model learns contextual codon harmonization, synthetic constructs should be experimentally validated and analyzed for extreme RNA secondary structure (e.g., strong 5' hairpins) prior to wet-lab synthesis.
• Sequence Length: Optimized for standard protein domain lengths. For exceptionally large multi-kilobase proteins, generation length boundaries should be monitored.
• Biosafety & Dual-Use Research: This model must only be utilized for benign research and industrial protein expression. Users must comply with international biosecurity standards (e.g., IGSC gene synthesis screening guidelines) and refrain from optimizing sequences derived from dangerous pathogens or controlled toxins.

---

Citation

@misc{coli_byt5_small_2026,
  author = {Dandekar, Aakash},
  title = {Coli-ByT5-Small: Byte-Level Sequence-to-Sequence Model for Escherichia coli Codon Optimization},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/aakashdandekar2006/Coli-ByT5-Small}}
}