VoiceMem Reply Adapter (Qwen3.6-35B-A3B, QLoRA v2)

Code 💻 / Default Models Env 🤗

[!Note] This repository contains a LoRA adapter only — not a standalone model. It is applied on top of Qwen/Qwen3.6-35B-A3B with peft. You must obtain the base model separately under its own license and access conditions.

[!Important] The repository name does not match its contents. This repo is named VoiceMem_SLM_Qwen25_omni, but the weights published here are the Qwen3.6-35B-A3B reply adapter, not a Qwen2.5-Omni checkpoint.

This is the reply model of VoiceMem — a memory system for real-time voice agents built on a streaming dual-brain architecture. VoiceMem retrieves memory while the user is still speaking; this adapter is the generation side, tuned to consume the retrieved memory and answer from it rather than ignore it or restate it verbatim.

Highlights

• Memory-grounded generation — trained on multi-turn SFT data where the context is Top-K retrieved memory, not raw conversation transcripts.
• Adapter-only, 180 MB — rank-32 LoRA over the language-model tower; the base model is untouched and separately licensed.
• Reproducible — the training script and default hyperparameters that produced checkpoint-3318 ship in the code repo (finetune/train.py).
• Drop-in with the VoiceMem stack — scripts/realtime_funasr_qwen.py runs mic → streaming memory prefetch → memory-grounded reply.

Model Overview

• Type: LoRA / PEFT adapter for a Causal Language Model
• Base model: Qwen/Qwen3.6-35B-A3B
• Training stage: QLoRA supervised fine-tuning
• Adapter
  • Rank (r): 32
  • Alpha: 64
  • Dropout: 0.05
  • Bias: none
  • Task type: CAUSAL_LM
  • Target modules: attention and FFN projections of the language-model tower (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, in_proj_*, out_proj, shared_expert_gate)
  • Trainable file size: 180 MB (adapter_model.safetensors)
• Training
  • Checkpoint: checkpoint-3318 (2 epochs, 3,318 global steps, seed 42)
  • Precision: bf16, gradient checkpointing on
  • Optimizer: adamw_torch_fused, lr 2e-4, cosine schedule, warmup ratio 0.03
  • Weight decay: 0.1, Adam betas (0.9, 0.95)
  • Batch: 8 per device × 2 gradient accumulation steps
  • Max sequence length: 2,048
• Data: multi-turn SFT JSONL. Training data are not released. Provenance, licenses, and consent status are documented in the code repository before any data release.

Quickstart

pip install transformers peft torch

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_id = "Qwen/Qwen3.6-35B-A3B"
adapter_id = "zhifeixie/VoiceMem_SLM_Qwen25_omni"

tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
    base_model_id, trust_remote_code=True, dtype="bfloat16", device_map="auto"
)
model = PeftModel.from_pretrained(base, adapter_id)
model.eval()

With the VoiceMem memory system

The adapter expects retrieved memory in its context. Use VoiceMem to produce it:

pip install voicemem funasr sounddevice
hf download zhifeixie/VoiceMem_Default_Models_Env --local-dir ./models

export OPENAI_API_KEY=sk-...          # write-side fact extraction only; retrieval stays local
python scripts/realtime_funasr_qwen.py

Train your own adapter

The defaults in the training script are the ones that produced checkpoint-3318; running it unchanged reproduces this adapter.

pip install trl peft datasets accelerate bitsandbytes
python finetune/train.py --data data/train.jsonl

Data format, VRAM requirements, and how to swap the base model: see finetune/README.md.

Evaluation

This adapter

On AudioMC INFERENCE_MEMORY (132 conversations, 233 rubric criteria), checkpoint-3318 satisfies 97 / 233 (41.6%) criteria. Under the same protocol, GPT-4o-mini satisfies 96 / 233 (41.2%) — an improvement of 1 criterion, or +0.43 percentage points. GPT-4o-mini is the judge for both runs.

[!Note] This is a small margin on a single benchmark. Treat it as evidence that the adapter does not regress against a strong baseline, not as evidence of a large capability gain.

The VoiceMem system

The numbers below measure the memory system, not this adapter in isolation. They are reproduced here for context; see the code repository for the protocol.

Files

Optimizer shards, DeepSpeed state, RNG state, scheduler state, and training logs are not model artifacts and are not published here.

Verify a download:

sha256sum -c checksums.sha256

Intended Use and Limitations

Intended use. Research on long-term conversational memory and memory-grounded answer generation.

Limitations.

• This is a research adapter. It has not been evaluated for safety-critical use and does not replace the base model's own safety documentation.
• It may reproduce biases and errors from both the training data and the base model.
• It is tuned for memory-grounded replies. Outside that setting it has no evaluated advantage over the base model.
• The adapter cannot verify the memories it is given. Wrong or stale retrievals produce confident wrong answers.
• Evaluation covers Chinese and English only.

License

The adapter is released under the Apache License 2.0. The base model carries its own license, which you must accept separately — this release grants no rights to it.

Citation

If you find this adapter useful, please cite the VoiceMem repository:

@software{voicemem,
    title  = {VoiceMem: Streaming Dual-Brain Memory for Real-Time Voice Agents},
    author = {{VoiceMem Team}},
    year   = {2026},
    url    = {https://github.com/lang-jiaqi/Voicemem_open}
}