leonsarmiento/Hy-Memory-30B-A3B-8bit-mlx

This model leonsarmiento/Hy-Memory-30B-A3B-8bit-mlx was converted to MLX format from tencent/Hy-Memory-30B-A3B using mlx-lm version 0.32.2 (custom shard-streaming quantizer — the 60 GB BF16 source exceeds 48 GB of unified memory).

Uniform 8-bit quantization, group size 64. ~8.500 bits-per-weight; norms and the per-layer fp32 expert_bias routing buffer are kept at full precision (standard mlx-lm convention).

About the source model

tencent/Hy-Memory-30B-A3B is Tencent Hunyuan's compact long-term-memory writer (~30B total / ~3B active, 48 layers = 1 dense + 47 MoE, 128 routed experts with 8 active + 1 shared, sigmoid routing with correction bias, 262K context). It executes the write path of external memory frameworks — deciding what to retain, how to represent it, and how new evidence updates a persistent memory bank. It directly replaces the general-purpose LLM inside Mem0, MemOS, and Hy-Memory, and transfers to unseen frameworks such as EverMemOS.

Source benchmarks (fixed 20% subsample, six metrics, writer swapped only):

Beats Claude Opus 5 (54.62), DeepSeek V4.1 Flash (53.64), and Qwen3.6-35B-A3B (50.37) as the writer. Technical report: arXiv 2610.22196.

⚠️ Requirements: custom hy_v3 model class

hy_v3 is a brand-new architecture — stock mlx-lm cannot load this repo. This build requires the companion model class:

1. Download hy_v3.py from this repo
2. Copy it into your mlx-lm install: cp hy_v3.py <site-packages>/mlx_lm/models/
3. Load normally (mlx-lm resolves model_type: hy_v3 by module name)

oMLX / LM Studio: not loadable — neither engine ships a hy_v3 class (as of 2026-10-16), so this repo will not open there. The supported serving path is mlx_lm.server with the custom class installed (see Serving below). The class's sanitize() stacks the checkpoint's individual expert tensors (mlp.experts.{i}.{gate,up,down}_proj) into mlx-lm SwitchGLU layout at load time (mixtral-style) and maps the checkpoint's own naming (mlp.router.gate, mlp.expert_bias, mlp.shared_mlp) — no remote code or extra plumbing needed.

Serving: mlx_lm.server only (no oMLX / LM Studio)

This model must be served with mlx_lm.server — there is no engine support for hy_v3 anywhere else. Once the custom class is installed (step above), serve it with a small launcher that sets the Metal memory limit and fails fast if the class is missing. Example launcher (from this build's validated serve_hy_memory_8bit.py):

#!/usr/bin/env python
import subprocess
import sys

import mlx.core as mx

mx.set_memory_limit(int(40 * 1024**3))  # peak ≈ 37.6 GiB: 29.7 weights + 6.4 KV + ~1.5 activations

# Sanity check: fail fast if the custom class is not installed
try:
    import mlx_lm.models.hy_v3  # noqa: F401
except ImportError:
    sys.exit("hy_v3 class missing — copy hy_v3.py from this repo into mlx_lm/models/")

cmd = [
    sys.executable, "-m", "mlx_lm.server",
    "--model", "/path/to/Hy-Memory-30B-A3B-8bit-mlx",
    "--host", "127.0.0.1",
    "--port", "6972",
    "--temp", "0.2",
    "--top-p", "0.95",
    "--min-p", "0.05",
    "--max-tokens", "32768",
    "--kv-bits", "8",
    "--kv-group-size", "64",
    "--prompt-cache-bytes", "7000000000",
    "--chat-template-args", '{"reasoning_effort": "no_think"}',
]
raise SystemExit(subprocess.call(cmd))

Why these flags:

• 8-bit KV cache (--kv-bits 8 --kv-group-size 64) — 4 KV heads × 128 head_dim × 2 (K+V) × 48 layers × 2 bytes ≈ 96 KiB/token at fp16 → ~12.9 GiB at a 131K context budget, which doesn't fit beside the ~29.7 GiB weights. 8-bit KV cuts full-context KV to ~6.4 GiB, near-lossless.
• 40 GiB Metal memory limit — with quantized KV the peak is ≈ 29.7 (weights) + 6.4 (KV) + ~1.5 (activations) ≈ 37.6 GiB. Budget one full-context session at a time; requests beyond ~131K tokens hit the ceiling.
• --prompt-cache-bytes 7000000000 — the prompt-cache byte cap evicts old sessions instead of accumulating a second cache.
• --chat-template-args '{"reasoning_effort": "no_think"}' — upstream's recommended default for the memory-writer task; append a later --chat-template-args occurrence to override (argparse takes the last one).

Endpoint: http://127.0.0.1:<port>/v1/chat/completions. Note that mlx_lm.server resolves the OpenAI request's model field against the loaded model id — send the full model path (as returned by /v1/models), not a short alias.

Recommended Parameters

Temperature: 0.0 Top P: 1.0 Reasoning: no_think (recommended by upstream for the memory-writer task)

The chat template supports thinking mode; disable it via tokenizer.apply_chat_template(messages, ..., reasoning_effort="no_think") (or pass chat_template_kwargs={"reasoning_effort": "no_think"} on an OpenAI-compatible server).

Use with mlx

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("leonsarmiento/Hy-Memory-30B-A3B-8bit-mlx")

prompt = "hello"

if tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True
    )

response = generate(model, tokenizer, prompt=prompt, verbose=True)