TransMem-qwen3-4B-D4
A TransMem memory-module checkpoint for Qwen3-4B, accompanying TransMem: Transforming Hidden States into Memory for Large Language Models.
Paper (OpenReview) · Paper (PDF) · GitHub / code and evaluation · arXiv
Overview
TransMem reuses sparse hidden states computed from a long context as memory. During answer decoding, independent memory modules transform these states into residual updates at selected backbone layers. A learned token-dependent gate controls each update. The backbone stays frozen.
The modules are trained with evidence-conditioned self-distillation: a teacher sees supporting evidence and the question, while a memory-augmented student sees the full context and learns to match the teacher's predictions. The paper's main configuration is trained on HotpotQA and evaluated on HotpotQA, LoCoMo, and MemoryAgentBench.
Checkpoint details
| Item | Value |
|---|---|
| File | best.pt |
| Backbone | Qwen/Qwen3-4B-Instruct-2507 |
| Insertion depth | D = 4 (Last-4) |
| Injection layers | 32–35 (zero-based) |
| Architecture | One independent single-layer Transformer memory module per injection layer |
| Memory segments | N = 4 |
| Dynamic gate | Token-scalar centered sigmoid, g = 2σ(z) |
| Added trainable parameters | 429,958,148 (429.958 M) |
| Backbone parameters | 4,022.468 M |
| Added parameters / backbone | 10.69% |
The parameter count includes all four memory modules, their output projections, normalization parameters, and dynamic gates. It excludes the frozen backbone. M means 1,000,000 parameters.
This repository contains the auxiliary module, rather than a complete standalone language model or a PEFT LoRA adapter. Download the matching backbone separately and use the native TransMem code to load best.pt; AutoModelForCausalLM.from_pretrained on this repository alone is insufficient. The checkpoint contains its module configuration and state dictionary.
Parameter comparison (Tables 1–2 backbones)
| Backbone (one row per backbone in Tables 1–2) | Backbone parameters (M) | Added TransMem parameters (M) | Trainable parameters (M) | Added parameters (% of backbone) |
|---|---|---|---|---|
| Qwen3-4B | 4,022.468 | 429.958 | 429.958 | 10.69% |
| Qwen2.5-14B-Instruct | 14,770.034 | 1,205.945 | 1,205.945 | 8.16% |
| Llama3.1-8B-Instruct | 8,030.261 | 939.591 | 939.591 | 11.70% |
The backbone is frozen; all added TransMem parameters, including the gates, are trainable. Percentages use exact counts before rounding. Counts were checked against the released checkpoint tensor shapes and the module architecture.
Download
pip install -U huggingface_hub
hf download Rayleihaodong/TransMem-qwen3-4B-D4 --local-dir checkpoints/TransMem-qwen3-4B-D4
hf download Qwen/Qwen3-4B-Instruct-2507 --local-dir models/Qwen3-4B
For gated backbones, first obtain access through their Hugging Face model page and authenticate with hf auth login.
Use with the official implementation
git clone https://github.com/Haodong-Lei-Ray/TransMem.git
cd TransMem
python -m venv .venv
source .venv/bin/activate
pip install torch transformers accelerate pandas pyarrow tqdm nltk rouge-score pytest
export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"
export MODEL_PATH=/absolute/path/to/the/matching/backbone
export CKPT=/absolute/path/to/TransMem-qwen3-4B-D4/best.pt
A CUDA-enabled PyTorch environment is recommended. The official evaluation code uses local_files_only=True for the backbone, so download it before evaluation.
For example, after preparing the official HotpotQA distractor evaluation JSON as described in the code repository:
CUDA_VISIBLE_DEVICES=0 python scripts/eval/eval_hotpot_official.py \
--data-file data/hotpotqa-benchmark/hotpot/hotpot_dev_distractor_v1_eval.json \
--model-path "$MODEL_PATH" \
--mode transmem \
--ckpt "$CKPT" \
--output-json eval_results/hotpot_transmem/shard_0.json \
--num-shards 1 \
--shard-id 0 \
--max-answer-tokens 50 \
--attn-impl sdpa
The native evaluator reads the layered architecture and gate configuration from the checkpoint. Follow the GitHub README for dataset preparation, LoCoMo protocol dependencies, MemoryAgentBench judging, baseline runs, and shard merging.
Paper results
Scores below are reported in Tables 1–2 of the supplied OpenReview manuscript; they are not measurements rerun for this model card.
| Benchmark / metric (%) | Frozen backbone | + TransMem |
|---|---|---|
| LoCoMo, overall F1 | 40.79 | 54.11 |
| HotpotQA, EM | 42.35 | 50.15 |
| HotpotQA, F1 | 56.00 | 66.20 |
| MemoryAgentBench, average score | 28.95 | 40.00 |
LoCoMo uses categories 1–4. MemoryAgentBench's average is the paper's reported aggregate; individual sources have distinct scoring protocols, and some require an external judge. Keep dataset splits, prompts, context budgets, and decoding settings aligned when comparing results. Earlier public paper versions contain different numbers.
Limitations and license
TransMem inherits the matching backbone's generation limitations and may produce incorrect answers. The added modules do not by themselves extend the backbone's native context window. Parameter overhead is distinct from runtime overhead: sparse memory computation and injection into only four layers determine the additional work.
This model repository declares the MIT license. The backbone, code, datasets, and paper have their own licenses; consult each source before use.
Citation
If you use this checkpoint or the TransMem method, please cite:
@misc{lei2026transmemtransforminghiddenstates,
title={TransMem: Transforming Hidden States into Memory for Large Language Models},
author={Haodong Lei and Junming Liu and Yirong Chen and Pinlong Cai and Botian Shi and Ding Wang and Hongsong Wang},
year={2026},
eprint={2607.29032},
archivePrefix={arXiv},
primaryClass={cs.MA},
url={https://arxiv.org/abs/2607.29032}
}
OpenReview manuscript: https://openreview.net/forum?id=FmBT4enjqy