Hy-Memory-30B-A3B-GGUF

Tencent Hunyuan

---

GGUF Quantization

This repository provides GGUF quantizations of Hy-Memory-30B-A3B for efficient local inference with llama.cpp.

The releases use Tencent Hunyuan's Hy-Memory quantization recipe: imatrix calibration plus MoE-aware mixed precision. Routed experts, which dominate parameter count, can be compressed aggressively, while shared experts, attention, and input/output layers keep higher precision. On the same fixed Mem0 LoCoMo setup, even the ~8 GB expert-mixed build stays within about 1.5 points of the strongest Q6_K result (57.25% vs 58.78%), so the quality drop remains small while the file size falls to roughly one quarter of Q8_0.

All six files passed llama.cpp --check-tensors, real no-think generation, and the complete fixed 20% Mem0 LoCoMo evaluation. Each version builds its own memory bank under identical settings. These are single-run scores: they do not claim lossless quantization, and lower-bit variants are not guaranteed to beat higher-bit ones.

Quantization Recipe

The recipe focuses on two design choices:

1. imatrix calibration collects activation statistics from calibration data to control quantization error.
2. MoE mixed precision assigns lower bit-widths to routed experts and keeps higher precision on shared experts, attention, and embedding / output layers. Expert count and routing structure are unchanged.

Encoding formats come from the open-source llama.cpp ecosystem (Q3_K_M, Q4_K_M, Q6_K, Q8_0, IQ1_M, IQ2_XXS). What this repository publishes is the Hy-Memory calibration and precision-allocation recipe on top of those formats. Five files use an importance matrix; Q8_0 is the standard 8-bit baseline without imatrix.

Spec Cheatsheet

Naming notes:

• K: K-quants block/sub-block scaling. Unrelated to MoE top-k routing.
• M / XXS: Medium / Extra Extra Small within each quantization family. Do not compare suffixes across families by name alone; IQ1_M actually uses fewer bits per weight than IQ2_XXS.
• IQ1_M / IQ2_XXS: two IQ encodings. For tensors of that type, average bit-widths are about 1.75 and 2.0625 bits/weight. The leading I is a letter, not the digit 1; these are not aliases of Q1_0 / Q2_0.
• imatrix: importance matrix from calibration activations; not a model size or bit-width.
• exps: routed experts compressed to the low-bit IQ type in that file.

File size still includes scales, floating-point leftovers, and metadata. Runtime VRAM also includes KV cache and framework overhead, so it is not equal to the download size.

Expert Mixed-Precision Allocation

Verified against GGUF tensor types:

Each mixed file has 141 routed-expert tensors in the corresponding IQ type, 338 other matrix tensors in Q8_0, and 287 tensors kept in F32. “Low-bit experts + Q8_0 elsewhere” is a precision allocation, not a 1-bit / 2-bit whole-model compress, and not expert pruning.

Community Ultra-Low-Bit Warning

Some community ultra-low-bit GGUF files can load successfully but fail memory writing: garbled text, endless repetition, or broken JSON. A Q1 / Q2 filename or a successful server start is not enough evidence that the model works.

On 2026-10-21 we checked vinnipukh/Hy-Memory-30B-A3B-GGUF at commit aa6b91554c36e108fe4463a20b5a8eac9121410f with matching SHA256 downloads, llama.cpp 5aeac37a35b44fde4bdffaec9159f2fcb253b622, H20, no-think, F16 KV, and no forced JSON decoding:

The check covered Chinese/English extract, update, delete, and fact-source prompts (temperature=0, max_tokens=256). All four community responses hit the token limit. Under the same prompts, local Q6_K returned valid JSON in all 8 trials. This is a basic output check, not a LoCoMo Judge score, and it does not isolate quantization error vs conversion vs backend issues. The conclusion applies only to the files and setup above.

By contrast, this repository's Q8_0-expsIQ1_M and Q8_0-expsIQ2_XXS use different IQ encodings, compress only routed experts to those low bit-widths, and have passed both structured-output checks and the full fixed 20% Mem0 LoCoMo pipeline. Distinguish releases by the full filename.

GGUF Evaluation Setup

• Harness: Mem0 2.0.19
• Benchmark: Fixed 20% LoCoMo test split
• Evaluation size: 2 users, 262 questions, and 59 memory chunks
• Retrieval: top_k=10, 10 turns per memory chunk
• Memory temperature: 0.7
• Reasoning mode: no-think
• Embedding model: qwen3-embedding-8b-for-online (4096 dimensions)
• Answer and judge model: gpt-5.6-sol
• Inference framework: official llama.cpp, commit 5aeac37a35b44fde4bdffaec9159f2fcb253b622
• KV cache: F16

All six releases completed 59 memory-chunk writes and 262 judged answers with no leftover evaluation errors. This directory does not include the BF16 Hugging Face weights.

Checksums

---

Model Introduction

Hy-Memory-30B-A3B is a compact long-term-memory writer model (~3B active parameters) developed by Tencent Hunyuan to execute the write path inside external memory frameworks. It directly replaces the general-purpose LLM inside Mem0, MemOS, and Hy-Memory (covering Chat, Working, and Coding scenarios), deciding what information to retain, how precisely to represent it, and how new evidence should update the persistent memory bank. The model is trained in two stages—Memory On-Policy Distillation (OPD) followed by Attributed Memory Reinforcement Learning—and transfers directly to unseen memory frameworks such as EverMemOS.

Cross-harness macro-average scores on the fixed 20% benchmark matrix. Each harness score averages six metrics equally; Overall averages the four harnesses equally. EverMemOS is held out from training.

---

Two-Stage Training

Figure 1: Hy-Memory-30B-A3B is trained inside the same deployed harness configurations in both stages. OPD supplies dense teacher supervision on student-written state chains; RL examines the resulting banks, assigns chunk-level credit, and carries the selected bank across windows. In the rollout grid, each cell is the bank after chunk t of rollout i, shaded by its chunk score s_i[t]. Advantages are normalized within each column (dashed: one chunk-level group), so the best rollout can differ from chunk to chunk. The outlined cell is the final bank with the highest sum_t s_i[t], which becomes the parent of the next window.

---

Experiments

The full cross-harness results below are reproduced from the original Hy-Memory-30B-A3B model card. GGUF-specific measurements are reported in the GGUF Quantization section above.

Evaluation Principles

The evaluation changes only the memory writer. For each comparison, the harness version and prompts are fixed, retrieval configuration is fixed, and the answer model, judge, and other framework-side components are held fixed. Throughout this report, every reference to a DeepSeek Flash endpoint denotes DeepSeek V4.1 Flash. Decoding settings are matched where APIs permit, each writer builds its own bank from an empty state, and test examples and scoring rules are identical.

For computational efficiency, every benchmark in the main four-harness matrix is evaluated on a fixed 20% subsample. Each harness score is the equal-weight average of six metrics: LoCoMo Judge, LongMemEval (LME), PersonaMem-v2 (PM-v2), Travel SPS, Shopping PS, and Code Memory Bench (CMB). CMB is itself the equal-weight average of two repository QA evaluations. The overall score is the equal-weight average of the four harness scores.

Final Cross-Harness Results

Table 1: Six-metric means on fixed 20% benchmark subsamples. The macro average weights the four harnesses equally.

Detailed Benchmark Results

Tables 2–5 report the six benchmark scores used to compute each harness mean in Table 1 on the same fixed 20% benchmark subsamples.

Table 2: Detailed results on the Mem0 harness

Table 3: Detailed results on the Hy-Memory harness

Table 4: Detailed results on the MemOS harness

Table 5: Detailed results on the held-out EverMemOS harness

---

Quickstart

We recommend using a recent official llama.cpp build. The commands below were validated with llama.cpp commit 5aeac37a35b44fde4bdffaec9159f2fcb253b622 and the Q6_K imatrix model.

The example uses Q6_K. For another release, change --model to the corresponding filename and keep --alias / the request model field consistent; keep the same no-think settings.

1. Launch the llama.cpp server

llama-server \
  --model ./Hy-Memory-30B-A3B-Q6_K.gguf \
  --alias Hy-Memory-30B-A3B-Q6_K \
  --host 127.0.0.1 \
  --port 8000 \
  --n-gpu-layers all \
  --flash-attn on \
  --ctx-size 65536 \
  --parallel 1 \
  --batch-size 2048 \
  --ubatch-size 512 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --temp 0.7 \
  --top-k 0 \
  --top-p 1.0 \
  --repeat-penalty 1.0 \
  --jinja \
  --chat-template-kwargs '{"reasoning_effort":"no_think"}' \
  --check-tensors

--ctx-size is the total context available to the server. The single-slot configuration above provides a 64K context window. If you use --parallel 2 to provide two slots, set --ctx-size 131072 to retain up to 64K context per request.

2. Call the server with the OpenAI Python SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="Hy-Memory-30B-A3B-Q6_K",
    messages=[
        {
            "role": "user",
            "content": (
                "Extract the durable memory from the following conversation and "
                "return only one concise sentence:\n\n"
                "User: I moved from Boston to Seattle in March 2025, and my "
                "preferred contact method is email."
            ),
        }
    ],
    temperature=0.7,
    top_p=1.0,
    max_tokens=512,
    extra_body={"reasoning_effort": "no_think"},
)

print(response.choices[0].message.content)

3. Call the server with curl

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Hy-Memory-30B-A3B-Q6_K",
    "messages": [
      {
        "role": "user",
        "content": "Extract the durable memory and return only one concise sentence: User: I moved from Boston to Seattle in March 2025, and my preferred contact method is email."
      }
    ],
    "temperature": 0.7,
    "top_p": 1.0,
    "max_tokens": 512,
    "reasoning_effort": "no_think"
  }'

The server-level --chat-template-kwargs enables no-think mode by default. Setting reasoning_effort=no_think in each request makes the behavior explicit and easier to audit.

---

Resource Availability

• Framework Code & Evaluation Utilities: https://github.com/Tencent-Hunyuan/Hy-Memory
• GGUF Model Weights and Model Card: https://huggingface.co/tencent/Hy-Memory-30B-A3B-GGUF
• Original BF16 Model: https://huggingface.co/tencent/Hy-Memory-30B-A3B
• Technical Report: https://arxiv.org/pdf/2610.22196