Hy-Memory-30B-A3B — MLX Mixed 4/6-bit

Mixed-precision MLX conversion of tencent/Hy-Memory-30B-A3B.

Hy-Memory is a memory writer, not a chat model. It reads a conversation and decides what is worth keeping — names, preferences, constraints, recurring facts — then returns them as clean JSON for an external memory framework (Mem0, MemOS, Hy-Memory). It is a 30B-parameter MoE with roughly 3B active per token, so it stays quick for its size while being far cheaper to run on the write path than a frontier generalist.

4.83 bits per weight · 17 GB · 100% extraction recall (70/70 facts)

Which repo to use

hy_v3 is not in MLX-LM upstream, and the two runtimes that can run it bring different adapters with different expert key names — so a checkpoint loads in exactly one of them.

Use this repo for MLX-LM. Use the -omlx repo for oMLX. Picking the wrong one fails loudly and immediately — each runtime rejects the other's layout with the same Received 423 parameters not in model: error.

Why mixed 4/6

The 4-bit base keeps the model small; the 6-bit bump protects the layers that actually matter for extraction quality. We measured it against the full 8-bit quant on the same hardware, and the practical difference is negligible:

That KL figure deserves context: sliced by how confident the 8-bit reference was, the two models agree 100% of the time wherever the reference was confident (top-1 probability > 0.9, 27% of positions). The divergence lives entirely in the tail of near-uniform distributions — positions where the reference itself is unsure. Only 3 of 2,040 positions disagreed while the reference was confident.

Quickstart

pip install -U mlx-lm

from mlx_lm import load, stream_generate

model, tokenizer = load("hermitdave/Hy-Memory-30B-A3B-MLX-Mixed4-6")

messages = [
    {
        "role": "system",
        "content": (
            "Extract explicit user facts and preferences from the conversation. "
            "Write each memory as a concise, self-contained statement. "
            "Do not infer or invent information. "
            'Return only a JSON object with a "memories" list of strings.'
        ),
    },
    {
        "role": "user",
        "content": (
            "My name is Alex. I live in Seattle and work as a software engineer. "
            "I mainly use Python and prefer concise answers with code examples. "
            "I'm vegetarian, so please keep that in mind when suggesting meals."
        ),
    },
]

prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_dict=False)
for r in stream_generate(model, tokenizer, prompt=prompt, max_tokens=512):
    print(r.text, end="", flush=True)

Output:

{
  "memories": [
    "Alex's name is Alex.",
    "Alex lives in Seattle.",
    "Alex works as a software engineer.",
    "Alex mainly uses Python.",
    "Alex prefers concise answers with code examples.",
    "Alex is vegetarian and requests meal suggestions to account for this."
  ]
}

Or serve it OpenAI-compatible:

mlx_lm.server --model hermitdave/Hy-Memory-30B-A3B-MLX-Mixed4-6 --port 8080

Recommended: temperature=0.0 for extraction — you want determinism, not creativity.

Quantization

Group size 64, affine mode. Allocation follows MLX-LM's mixed_4_6 recipe (the same logic as llama.cpp's Q4_K_M):

Layers receiving 6-bit: [0–5, 42–47, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35, 38, 41].

The weights are converted layer-by-layer from the bf16 source rather than by loading the whole model, because a 56 GB bf16 checkpoint does not fit alongside a quantized copy on a 64 GB machine.

Notes

• Custom architecture. hy_v3 (HYV3ForCausalLM) is not in MLX-LM, so hy_v3.py ships with this repo. MLX-LM finds it through the "model_file": "hy_v3.py" key in config.json — the only hook MLX-LM offers for a custom model class. Note that MLX-LM's --trust-remote-code flag is tokenizer-only and does not load a custom model class; without model_file, MLX-LM raises ValueError: Model type hy_v3 not supported. and never opens hy_v3.py at all. Inspect the file before running if you have concerns — it is 239 lines and does nothing but define the model.
• Context length is 262,144 tokens. Memory writes do not need it, but it is there.
• Two config corrections were required: the upstream moe_intermediate_size: 1536 contradicts the checkpoint's own expert weights (all 768-wide), and rope_theta moved into a nested rope_parameters block. Both are fixed in this repo's config.json.
• This model is a writer, not a retriever or answerer. It pairs with a general-purpose LLM, not replaces one.

Attribution

• Base model: tencent/Hy-Memory-30B-A3B by Tencent Hunyuan, Apache-2.0.
• hy_v3 MLX adapter: originally authored by QwQbb for the Hy-MT2 conversion; reused here with a stop_gradient fix on the router indices so the architecture is trainable.
• Conversion, quantization, and benchmarking: Hermes Agent using mlx-lm 0.31.3.
• Paper: arXiv:2610.22196

This is a community conversion, not an official Tencent release.

Citation

@misc{hy-memory-30b-a3b,
  title  = {Hy-Memory-30B-A3B},
  author = {Tencent Hunyuan},
  year   = {2026},
  eprint = {2610.22196},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url    = {https://arxiv.org/pdf/2610.22196},
}

License

Apache-2.0, same as upstream.