Gengram: Retrieval-Augmented Genomic Foundation Models

1. Introduction

Gengram is a novel conditional memory module designed for genomic foundation models (GFMs) that introduces explicit motif memory retrieval to enhance Transformer-based DNA sequence modeling. Unlike traditional GFMs that rely on dense computation to implicitly infer multi-nucleotide motifs, Gengram provides an efficient lookup mechanism for biological patterns through a genomic-specific hashing scheme.

Figure 1 illustrates the overall architecture of Gengram, together with the evaluation pipeline used to assess its effectiveness across multiple genomic benchmarks.

✨ Key Features

• 🎯 Explicit Motif Memory: Stores and retrieves k-mers (k=1-6) via hash-based lookup tables
• 🧬 Local Window Aggregation: 21bp window mechanism aligned with DNA helical structure
• ⚡ Computational Efficiency: Linear time complexity with minimal overhead
• 🔧 Architecture Agnostic: Compatible with various attention mechanisms (MHA, GQA, MLA)
• ⚖️ Stable Training: Improves load balancing in Mixture-of-Experts models
• 🔍 Biological Interpretability: Learns meaningful motif representations

✨ Biological Interpretability

Gengram exhibits clear biologically grounded behaviors, including:

• Reverse-complement symmetry in memory embeddings
• Context-dependent gating aligned with functional regions
• Hierarchical representation from shallow to deep layers

2. Model Information

Model Configuration

The following details the model configuration, including the parameterization of Gengram, MoE routing strategies, and training hyperparameters used across all experiments.

• Gengram Parameters

  These parameters control how Gengram operates within the Transformer layers, including which layers to apply it to, the n-gram sizes, and embedding dimensions.

• Mixture of Experts (MoE)

  These parameters define the Mixture-of-Experts architecture, including the number of experts, routing top-k, and load balancing strategies during training.

• Training Parameters

  These parameters specify the training setup, including sequence length, batch sizes, precision, and attention optimizations.

Pre-training Data

• Human Sequences: HPRC Release 2, GRCh38, CHM13
• Non-human Primates: NCBI RefSeq database
• Total: 200B tokens (8k context) + 100B tokens (32k context)

3. Performance Evaluation

Gengram demonstrates strong performance across multiple genomic benchmarks, achieving competitive results despite being trained on significantly fewer tokens and with a smaller model size.

• Key Observations

  • Data Efficiency: Achieves comparable performance using ~10×–40× fewer tokens
  • Motif-Dominated Tasks: Up to 14% improvement
  • Long-Context Modeling: Enhanced performance with shorter sequences
  • Training Efficiency: Better parameter utilization and stable MoE training

• Evaluation Benchmarks

  • Genomic Benchmarks (GB)
  • Nucleotide Transformer Benchmarks (NTB)
  • Long-Range Benchmarks (LRB)
  • Genos Benchmarks (GeB)

4. Quickstart

Model Download

Gengram model is available for download from Hugging Face. We provide torch version.

Pre-training

Run the pre-training script with the following command:

cd Gengram
bash Gengram_layer3-6-10_win21_pp2.sh

5. License

This repository and the Gengram model weights are licensed under the Apache License 2.0.

Please note that the primary use of Gengram model is to support genomics research, providing researchers with advanced analytical capabilities and long-context modeling tools powered by large-scale foundation models for the human genome. It is not intended for use in any manner that violates applicable laws or regulations, nor for any activities prohibited by the license agreement.

6. Citation and Acknowledgements

We acknowledge the high-quality sequencing data provided by CycloneSEQ, which forms an important foundation for this work. We also appreciate the inspiration from DeepSeek's Engram module and the framework support provided by Megatron-LM. Model training was conducted on the 021 Science Foundation Model and Zero2X open platform.

If you use this work in your research, please cite the following paper:

@article@article{gengram2026,
  title={Beyond Conditional Computation: Retrieval-Augmented Genomic Foundation Models with Gengram},
  author={Genos Team and Xu, Huinan and Feng, Xuyang and Chen, Junhong and Liu Junchen and Deng, Kaiwen and Ding, Kai and Long, Shengning and Shuai, Jiaxue and Li, Zhaorong and Liu, Shiping and Xue, Guirong and Xiao, Zhan},
  journal={arXiv preprint arXiv:2601.22203},
  year={2026}
}

7. Contact

For project-related questions, please open an issue. You can also contact the Genos Team at Genos@genomics.cn.