Memorilla checkpoints (Qwen3-8B)

Memory modules for Memorilla. All were trained with the frozen Qwen3-8B decoder and the frozen Qwen3-Embedding-4B encoder, and are used with the same two models.

Each folder holds memory.pt (bfloat16 weights, 194M parameters) and config.json (16 memory tokens, no document self-attention).

Loading

Install the code from GitHub, then download the checkpoints:

hf download memorilla/Memorilla-Qwen3-8B --local-dir checkpoints

from memorilla import MemoryModule

memory = MemoryModule.from_pretrained("checkpoints/stage3")

To evaluate a checkpoint, or to continue training from one with --init_memory:

scripts/evaluate.sh checkpoints/pv4 pv4
scripts/train.sh configs/stage3_multitask.yaml --init_memory checkpoints/stage2

The training guide describes the stages and recipes.

Citation

@inproceedings{fallahpour2026memorilla,
  title     = {Memorilla: Latent Semantic Memory for {LLM}s},
  author    = {Adibvafa Fallahpour and Parsa Idehpour and Vignesh Kothapalli and Nikita Mounier and Shirley Wu and Shayan Pardis and Jure Leskovec},
  booktitle = {COLM Workshop on Context Beyond the Window: Persistent Knowledge in Language Models},
  year      = {2026},
  url       = {https://openreview.net/forum?id=VV0vvzL783}
}