Xing4.0-29B-A4B-6bit-MLX — 6-bit MLX quant of Xing4.0-29B-A4B

Unofficial Apple Silicon quantization of XingChen-AGI/Xing4.0-29B-A4B, produced with mlx_lm.convert (MLX affine quantizer, group size 64) on an Apple M5 Max / 128 GB.

I am not affiliated with China Telecom AI. All upstream weights, benchmarks and license terms belong to them, and the upstream Apache-2.0 license governs this repository too (see License).

Format note: these are MLX safetensors, not GGUF. They will not load in llama.cpp / Ollama / LM Studio, and they will not load in PyTorch/vLLM/SGLang either. Use an MLX runtime.

Before you download: you need xing4_0 support in mlx-lm

Xing4.0 uses a new architecture (model_type: xing4_0) that mlx-lm does not implement yet. Without it any MLX runtime stops with:

ValueError: Model type xing4_0 not supported.

An implementation exists and is verified against the upstream PyTorch code (see Provenance), but it is not merged into mlx-lm at the time of writing. Until it is, these weights will not load anywhere. If you need them now, open an issue here and I will point you at the model file.

Pick a variant

All three drop the multi-token-prediction layer (see below). Throughput is nearly identical across the three because only ~4B parameters are active per token; the difference shows up in memory, not speed. Task-level accuracy after quantization has not been measured. Runtime memory also depends on context length and KV cache.

What is inside (read from the shipped config.json)

Quantization recipe

• mlx_lm.convert -q --q-bits 6 --q-group-size 64, mode affine, source BF16
• 476 modules quantized: attention projections, all 64 routed experts + shared expert per MoE layer, the dense FFNs of layers 0–1, embed_tokens and lm_head
• effective 6.512 bits per weight (scales and biases included)
• never quantized: all RMSNorms, the MoE router (mlp.gate), the hyper-connection tables (hc_fn, hc_base in BF16), and hc_scale / e_score_correction_bias kept in FP32

About the dropped MTP layer

The upstream checkpoint carries a 41st layer (1.71 B parameters) implementing multi-token prediction: eh_proj, enorm, hnorm, its own embed_tokens, a full attention + MoE block and a shared_head. MLX has no speculative-decoding path for this architecture, so those tensors are dropped and num_nextn_predict_layers is set to 0 to keep the shipped config self-consistent. If you want MTP, use the upstream BF16 checkpoint with a runtime that supports it.

Requirements

• Apple Silicon, macOS 15+ (built and smoke-tested on M5 Max, 128 GB)
• mlx-lm with xing4_0 support — see the warning above
• roughly 22.4 GB of free unified memory for a short prompt, more for long context

Usage

from mlx_lm import load, generate

# the custom tokenizer is loaded from the repo, so both flags are needed
model, tokenizer = load(
    "TokenAI-zer/Xing4.0-29B-A4B-6bit-MLX",
    tokenizer_config={"trust_remote_code": True},
    trust_remote_code=True,
)

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain Sinkhorn normalization in one sentence."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512, verbose=True))

mlx_lm.load forwards trust_remote_code to the model but not to the tokenizer, which is why tokenizer_config carries its own flag. Without it you get an unrelated-looking AttributeError: 'PreTrainedConfig' object has no attribute 'max_position_embeddings'.

The chat template supports enable_thinking (on by default) and tool calls. The model tends to write its reasoning trace in Chinese even for English prompts; that is upstream behaviour, not a quantization artifact.

oMLX

oMLX 0.6.4 cannot run this architecture yet, and its oQ mixed-precision quantizer fails on it for the same reason (its sensitivity pass cannot load the model). These are plain MLX quants, not oQ builds.

Recommended sampling

Upstream recommends, and the shipped generation_config.json matches:

Note that mlx_lm.generate does not apply a repetition penalty unless you pass a logits processor.

Provenance

The MLX implementation used to produce and load these weights was validated before quantizing:

Two upstream bugs found along the way, neither affecting these weights: the reference _init_weights initializes module.fn/base/scale while the class defines hc_fn/hc_base/hc_scale (random init from config fails, loading pretrained weights is unaffected), and mlx_lm.load does not forward trust_remote_code to the tokenizer.

Benchmarks

I publish no numbers I have not measured myself. The table below is upstream's, measured on the BF16 model, and is not a measurement of these quantized weights:

Measurements and issue reports ("quant X broke task Y") are welcome and will be merged into this table.

Known caveats

• Quantization is lossy. If you see a regression, compare against a higher-precision variant and the BF16 source before filing a bug.
• The hyper-connection mixing runs in the residual path of every layer and is kept in BF16/FP32 here; its sensitivity to weight quantization elsewhere in the model has not been studied.
• 262k context is the architecture's limit, not a promise: keep the KV cache inside your memory budget or the machine swaps.
• No MTP head, so no self-speculative decoding.
• Agentic and long-context behaviour at 6-bit is untested.

License

Distributed under the Apache License 2.0, inherited from XingChen-AGI/Xing4.0-29B-A4B. See LICENSE-NOTICE.md in this repository.

Citation

@misc{xing4-29b-a4b-mlx-6bit,
  title        = {Xing4.0-29B-A4B-6bit-MLX: MLX 6-bit quantization of Xing4.0-29B-A4B},
  author       = {TokenAI-zer},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/TokenAI-zer/Xing4.0-29B-A4B-6bit-MLX}},
  note         = {Unofficial quantization of XingChen-AGI/Xing4.0-29B-A4B}
}

@misc{liu2025trainingreporttelechat3moe,
  title         = {Training Report of TeleChat3-MoE},
  author        = {Xinzhang Liu and others},
  year          = {2025},
  eprint        = {2512.24157},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2512.24157}
}

Acknowledgements

• China Telecom AI (XingChen-AGI) for Xing4.0-29B-A4B and the mHC architecture.
• Apple MLX team for mlx and mlx-lm, whose DeepSeek-V3 implementation this architecture builds on directly.