Qwen-Coli

A Specialized Language Model for Context-Aware Codon Optimization in Escherichia coli

Qwen-Coli is a specialized language model for Codon Optimization for E. coli fine-tuned on top of Qwen2.5-Coder-1.5B-Instruct. The fine-tuned LoRA weights have been completely fused into the base model's 16-bit (bfloat16) transformer weights.

Because this repository contains the complete merged weights (model.safetensors), it does not require PEFT or a separate base model to run. It is directly ready for standard inference across the Hugging Face ecosystem.

For a local Gradio streaming web interface to test this model, see Qwen-Coli-1.5B-Gradio.

---

At a Glance

---

Overview & Description

Qwen-Coli bridges large-scale code intelligence with computational molecular biology. In synthetic biology, DNA functions as software code executed inside a living cell runtime. Just like software compilers optimize high-level source code to run efficiently on specific computer processor architectures, biological sequence optimizers must tailor genetic code to execute efficiently within a chosen host organism.

By adapting the advanced syntactic reasoning of Qwen2.5-Coder-1.5B-Instruct, Qwen-Coli performs intelligent reverse translation, sequence engineering, and context-aware codon optimization tailored for Escherichia coli (E. coli)—the most widely used production organism in biotechnology.

---

Intention Behind Training the Model

1. The Shortcomings of Classical Heuristics

For decades, computational codon optimization has relied on static lookup tables, primarily driven by maximizing the Codon Adaptation Index (CAI) or applying greedy "one amino acid $\to$ most frequent codon" replacement. While theoretically straightforward, this heuristic frequently fails in wet-lab experiments due to several fundamental biological oversights:

1. tRNA Pool Exhaustion & Ribosome Stalling: Over-optimizing every occurrence of an amino acid to the single most abundant codon creates acute local shortages in corresponding charged aminoacyl-tRNA pools. This results in translational pauses, ribosome queuing (traffic jams on the mRNA), and increased rates of amino acid misincorporation.

2. Disruption of Co-Translational Protein Folding Kinetics: Proteins do not fold only after synthesis is complete; nascent peptide chains begin folding hierarchically as they emerge from the ribosome exit tunnel. Natural endogenous genes use strategically positioned "slow" or sub-optimal codons at domain boundaries, hinges, and loop junctions to throttle elongation speed. This grants the nascent domain time to achieve its native tertiary conformation before downstream sequence is extruded. Naively maximizing translation speed eliminates these kinetic checkpoints, frequently precipitating misfolded proteins into insoluble inclusion bodies.

3. Ignoring 5' mRNA Secondary Structure Thermodynamics: In E. coli, translation initiation—rather than elongation—is predominantly the rate-limiting bottleneck. Stable secondary structures (hairpins with low Gibbs free energy $\Delta G$) around the 5' Translation Initiation Region (TIR, covering ~-30 to +30 nt relative to the start codon) can occlude the Shine-Dalgarno (SD) sequence and the start codon (AUG). This prevents the 30S ribosomal subunit from binding, yielding little to no protein regardless of how high downstream CAI is.

4. Inadvertent Introduction of Deleterious Sequence Motifs: Static replacement algorithms routinely introduce unintended biological artifacts:

   • Internal Shine-Dalgarno-like sites (e.g., AGGAGG or close variants) that cause internal ribosome binding and translational frameshifting.
   • Premature transcription termination motifs (rho-independent stem-loops followed by poly-U tracts).
   • Cryptic promoter sequences (-10 / -35 hexamers) that cause spurious internal transcription.
   • Conflicting restriction enzyme sites (e.g., BsaI, BsmBI, EcoRI, BamHI) that break downstream cloning and Golden Gate assembly workflows.

2. Why a Code Language Model?

Software code models like Qwen2.5-Coder are pre-trained to resolve rigid syntactic constraints, preserve semantics across variable representations, and manage complex contextual interdependencies. When applied to biological coding sequences:

• The model treats the amino acid sequence as an immutable semantic invariant while navigating synonymous nucleotide substitutions.
• Long-range attention heads capture context across several hundred codons simultaneously, balancing global GC distribution and local elongation rhythm.
• The model generates nucleotide sequences that strike an optimal balance between translational efficiency, soluble folding yields, and cloning compatibility.

---

Codon Optimization in Escherichia coli (E. coli)

Biological Context & Industrial Relevance

Escherichia coli is the premier expression chassis for molecular biology and biotechnology, powering the production of:

• Biopharmaceuticals & Therapeutics: Recombinant human insulin, human growth hormone (hGH), interferons, and cytokines.
• Next-Generation Antibodies: Single-domain antibodies (nanobodies / VHH), Fab fragments, and scFvs.
• Industrial Enzymes: Biocatalysts for green chemistry, polymer breakdown, and food processing.
• Structural Biology: High-yield targets for Cryo-EM, X-ray crystallography, and NMR spectroscopy.

When expressing heterologous genes sourced from humans, plants, fungi, or viruses, dramatic differences in evolutionary codon usage regularly cause low or undetectable expression titers unless sequences are thoroughly optimized.

The Genetic Code & Codon Usage Bias (CUB)

The genetic code comprises 64 triplet codons that map to 20 canonical amino acids and 3 termination signals (Amber UAG, Ochre UAA, Opal UGA). Because most amino acids are specified by 2 to 6 synonymous codons, the code is degenerate.

Different organisms exhibit distinct Codon Usage Bias (CUB) reflecting cellular isoacceptor tRNA concentrations. In E. coli, highly expressed genes (e.g., elongation factor tufA, RNA polymerase subunits, ribosomal protein operons) heavily favor specific codons that match the most abundant tRNA species.

The "Rare Codon" Bottleneck

Heterologous sequences frequently contain codons that correspond to critically low tRNA pools in E. coli. The most problematic rare codons include:

Clusters of rare codons (e.g., AGG-AGA or ATA-CTA) trigger ribosome pausing, which exposes the unshielded mRNA to rapid cleavage by host endonucleases (such as RNase E), drastically reducing mRNA half-life and usable protein yield.

Core Optimization Dimensions Addressed by Qwen-Coli

1. Balanced Codon Selection (Harmonization): Selects codons compatible with E. coli tRNA abundance while preserving physiological translation dynamics.
2. GC Content Balancing: Maintains optimal whole-gene GC composition (~48% to 54%), eliminating extreme GC or AT islands that hinder PCR amplification, gene synthesis, or transcription.
3. 5' Initiation Region Relaxation: Minimizes RNA secondary structure stability around the start codon to ensure open access for the 30S ribosomal pre-initiation complex.
4. Deleterious Motif Screening: Automatically circumvents internal promoters, internal Shine-Dalgarno traps, and assembly restriction sites (e.g., BsaI, BsmBI, SapI, EcoRI).
5. Support for Common Strains: Aligns with expression systems across BL21(DE3), K-12, Rosetta, Origami, and DH5α.

---

Model Details

• Model Name: Qwen-Coli
• Model Type: Full Causal Language Model (Merged Standalone)
• Base Model: Qwen/Qwen2.5-Coder-1.5B-Instruct
• Architecture: Qwen2ForCausalLM
• Total Parameters: ~1.54 Billion
• Precision: bfloat16 (model.safetensors ~3.09 GB)
• Context Length: 32,768 tokens
• Attention Architecture: Grouped-Query Attention (GQA: 12 Query heads, 2 Key/Value heads)
• Layers: 28 transformer blocks (Hidden Size: 1536, Intermediate Size: 8960)
• Vocabulary Size: 151,936
• License: Apache 2.0
• Primary Domain: Computational Biology / Synthetic Biology / E. coli Expression

---

Default Generation Parameters

Configured in generation_config.json:

---

Getting Started

1. High-Level Pipeline (transformers)

from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="aakashdandekar2006/Qwen-Coli",  # or local path
    torch_dtype="bfloat16",
    device_map="auto"
)

messages = [
    {
        "role": "system",
        "content": (
            "You are Qwen-Coli, an expert computational biologist specialized in codon optimization "
            "and recombinant protein expression in Escherichia coli."
        )
    },
    {
        "role": "user",
        "content": (
            "Optimize the following amino acid sequence for high-level recombinant expression in E. coli BL21(DE3). "
            "Ensure rare codons (such as AGG, AGA, CTA, ATA, CCC) are avoided, maintain balanced GC content (~50-52%), "
            "eliminate internal Shine-Dalgarno motifs, and output the optimized DNA sequence in FASTA format:\n\n"
            "MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVTTFSYGVQCFSRYPDHMKQHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNTPIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITHGMDELYK"
        )
    }
]

output = generator(messages, max_new_tokens=1024)
print(output[0]["generated_text"][-1]["content"])

---

2. Standard Inference (AutoModelForCausalLM)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "aakashdandekar2006/Qwen-Coli"  # or local path

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model.eval()

# Example: Reverse-translating a nanobody / therapeutic peptide with Golden Gate cloning constraints
messages = [
    {
        "role": "system",
        "content": (
            "You are Qwen-Coli, an expert synthetic biology assistant. You optimize coding sequences "
            "for heterologous expression in Escherichia coli while taking into account codon usage bias, "
            "tRNA abundance, mRNA secondary structure, and restriction enzyme avoidance."
        )
    },
    {
        "role": "user",
        "content": (
            "Reverse-translate and optimize the following therapeutic nanobody sequence for expression in E. coli. "
            "Avoid internal BsaI (GGTCTC) and EcoRI (GAATTC) recognition sites for Golden Gate assembly compatibility:\n"
            "Sequence: EVQLVESGGGLVQPGGSLRLSCAASGFTFSSYAMSWVRQAPGKGLEWVSAISGSGGSTYYADSVKGRFTISRDNSKNTLYLQMNSLRAEDTAVYYCAK"
        )
    }
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([prompt], return_tensors="pt").to(model.device)

# Generate response
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=0.7,
        top_p=0.8,
        top_k=20,
        repetition_penalty=1.1,
        do_sample=True,
        pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id
    )

input_len = inputs.input_ids.shape[1]
response = tokenizer.decode(outputs[0][input_len:], skip_special_tokens=True)
print(response)

---

Benchmark & Validation: Wild-Type GFP Case Study

To validate sequence fidelity, reading-frame preservation, codon adaptation dynamics, and motif clearance, Qwen-Coli was evaluated on the canonical wild-type Aequorea victoria Green Fluorescent Protein (wtGFP, 238 aa; UniProt: P42212) against a traditional greedy optimizer baseline (which naively substitutes each amino acid with the single most frequent synonymous codon in E. coli).

=================================================================
                    BENCHMARK RESULTS (GFP — 238 aa)
=================================================================
Target length: 238 aa (714 nt expected)
-----------------------------------------------------------------
Metric                    | Qwen-Coli (LLM)   | Traditional Greedy
-----------------------------------------------------------------
Translation Check         | PASS (100% ID)    | PASS (100% ID)
Generated Length (nt)     | 714               | 714
Codon Adaptation (CAI)    | 0.809             | 1.000
GC Content (%)            | 49.6%             | 48.6%
Restriction Sites (Panel) | 0                 | 2
-----------------------------------------------------------------
Sequence translated successfully with zero amino acid mismatches.
First 60 nucleotides:
  Model:  ATGTCCAAAGGGGAAGAATTATTCACCGGCGTGGTGCCAATCCTCGTAGAACTGGATGGC...
  Greedy: ATGAGCAAAGGCGAAGAACTGTTTACCGGCGTGGTGCCGATTCTGGTGGAACTGGATGGC...
=================================================================

Biological & Computational Analysis

1. Translation Fidelity (100% Identity):

   • Result: PASS. Zero missense or nonsense mutations across all 238 residues.
   • Reference Check: The generated 714 nt sequence was translated in silico using the standard bacterial genetic code (NCBI Translation Table 11) and aligned against the 238-residue wild-type GFP reference (MSKGEELFTG...). Every codon precisely preserves the intended protein primary sequence with zero reading-frame slippage or premature truncation.

2. Sequence Length & Frame Integrity:

   • Result: Exactly 714 nucleotides ($238 \times 3\text{ nt}$), strictly matching the open reading frame (ORF) without insertions, deletions, or downstream frame corruption.

3. Balanced Codon Adaptation Index (CAI = 0.809):

   • Traditional greedy optimization maximizes CAI to 1.000 by universally selecting the highest-frequency codon for every residue.
   • While high CAI correlates with translational efficiency, literature suggests that extreme bias toward sole codons can risk taxing specific aminoacyl-tRNA pools under high expression induction. Furthermore, native genes frequently utilize synonymous codon variations and subtle kinetic pauses at structural boundaries to facilitate co-translational folding.
   • Qwen-Coli achieved a CAI of 0.809, reflecting a balanced physiological adaptation profile typical of robustly expressed genes while maintaining synonymous diversity. (Note: specific folding kinetics, solubility fractions, and in vivo expression titers are protein-dependent hypotheses that require experimental wet-lab validation).

4. Optimal GC Content (49.6%):

   • Sits right in the center of the physiological sweet spot for E. coli (45%–55%), supporting mRNA secondary structure stability while avoiding extreme GC hairpins or AT-rich instability tracts.

5. Restriction Site Clearance (0 vs. 2 Forbidden Sites):

   • Screened Enzyme Panel: Checked against common Type IIS Golden Gate assembly enzymes (BsaI GGTCTC, BsmBI CGTCTC, SapI GCTCTTC) and classical cloning/BioBrick endonucleases (EcoRI GAATTC, BamHI GGATCC, HindIII AAGCTT, XbaI TCTAGA, PstI CTGCAG).
   • Result: Qwen-Coli introduced 0 restriction sites from the screened panel, contextually navigating synonymous codon choices to keep the gene synthesis- and cloning-ready.
   • In contrast, the traditional greedy baseline introduced 2 forbidden restriction sites, which would require manual silent mutagenesis prior to cloning.

[!NOTE] Scope of Evaluation: Evaluating wild-type GFP provides a concrete proof-of-concept and sanity check for reading-frame fidelity, balanced CAI, and restriction clearance. However, a single protein sequence does not constitute an exhaustive benchmark. Systematic multi-protein benchmarking across diverse protein classes (e.g., membrane proteins, multi-domain complexes, GC-biased heterologous targets, and low-solubility enzymes) alongside physical wet-lab expression assays is part of ongoing evaluation.

---

Prompting Structure & Best Practices

Qwen-Coli is an instruction-tuned model. When interacting with the model via Hugging Face transformers, simply use standard conversational role messages with tokenizer.apply_chat_template():

messages = [
    {
        "role": "system",
        "content": "You are Qwen-Coli, a specialized assistant for E. coli codon optimization and biological sequence design."
    },
    {
        "role": "user",
        "content": "Optimize the following sequence for E. coli BL21(DE3) with balanced GC content (~50-52%): <YOUR_SEQUENCE>"
    }
]

Key Recommendations for Optimal Results:

1. Specify Target Host / Strain: Explicitly state your host strain (e.g., BL21(DE3), K-12 MG1655, Rosetta(DE3), DH5α).
2. State Cloning & Restriction Constraints: Mention any Type IIS or standard restriction enzyme recognition sites to prohibit (e.g., "Exclude BsaI (GGTCTC), BsmBI (CGTCTC), EcoRI (GAATTC), and BamHI (GGATCC) for Golden Gate assembly").
3. Set Thermodynamic & Composition Goals: Request specific GC boundaries (e.g., "Maintain GC content between 49% and 53%") or low 5' TIR secondary structure stability for translation initiation.
4. Define Output Formatting: Request outputs in FASTA format, annotated codon tables, or clean nucleotide sequences for direct submission to commercial DNA synthesis providers.

---

Limitations and Ethical Considerations

• In Silico & Experimental Validation: Computational sequence designs generated by Qwen-Coli should always be verified in silico (e.g., verifying 100% translated protein identity using Biopython or BLAST) and evaluated experimentally in wet-lab expression trials.
• Biosecurity & Responsible Science: Users must ensure that sequence design activities strictly adhere to applicable international biosecurity frameworks, including the Guidance for Providers of Synthetic Double-Stranded DNA and relevant Select Agents and Toxins regulations.
• Model Hallucinations: Autoregressive language models can occasionally propose non-canonical sequence motifs or inaccurate thermodynamic parameters. Critical cloning constraints must be independently validated before physical gene synthesis.
• License: Distributed under the Apache 2.0 license.