NVIDIA-Nemotron-3.5-Super-120B-A12B-q3-mlx

English | 繁體中文

Text-only MLX conversion of NVIDIA Nemotron 3.5 Super (120B total / 12B active, hybrid Mamba-Transformer latent MoE) for Apple Silicon, packaged so the routed experts stay on disk and are streamed on demand. All 40 MoE layers' routed experts are 3-bit; the resident trunk is mixed precision (see below). There is no ternary weight in this checkpoint.

This is not a standard mlx-lm checkpoint. mlx_lm.load() cannot open it (checked with mlx-lm 0.32.2: it fails with FileNotFoundError: No safetensors found, because the weights live in trunk/ and experts/, not in model*.safetensors). Use the included runtime nemo_offload.py.

Provenance

What is in this repo

How to run

Requirements: Apple Silicon Mac, macOS, Python ≥ 3.10, and enough free disk for 53.4 GiB. Fast internal SSD recommended.

pip install -r requirements.txt          # mlx, mlx-lm (for models/nemotron_h.py), transformers, numpy
hf download Jakevin/NVIDIA-Nemotron-3.5-Super-120B-A12B-q3-mlx --local-dir ./nemotron-q3
python ./nemotron-q3/nemo_offload.py --model-dir ./nemotron-q3 --prompt "Explain mixture-of-experts in two sentences."
python ./nemotron-q3/nemo_offload.py --model-dir ./nemotron-q3 --interactive

Options: --max-tokens, --temp (0 = greedy, default), --top-p, --thinking (chat template enable_thinking=True), --expert-cache-gb (default: everything the memory rule allows), --prefetch (cross-layer router prefetch; off by default, it was slower in our test).

How it works: the trunk is loaded into unified memory; each MoE layer gets a fixed number of expert slots (LRU). On a miss the expert rows are read from SSD with os.preadv straight into the slot buffer. Prefill reads only the experts the prompt actually uses. Memory budget = 0.8 × (free + purgeable + file-backed pages), capped at Metal's recommended working set; a guard aborts the process if swap grows by more than 8 GiB.

Measured results (read the caveats)

All numbers below are our own measurements on one Mac mini, Apple M4 Pro, 64 GB unified memory, macOS 26.6.2, experts read from an external USB SSD, using mlx 0.32.4 / mlx-lm 0.32.2. Source: our run log RESULTS_remedy.md (2026-10-18).

Quality vs. the FP8 reference — engineering pilot only

• Reference (REF): the RedHatAI FP8 checkpoint, dequantized and computed in BF16, same tokens.
• Data: 8 short dev sequences (code, English essay, math, Chinese chat), 1,457 token positions, teacher-forced. These 8 sequences had already been seen/used during development (not held-out). No confidence intervals. This is an engineering sanity check, not a benchmark and not evidence of downstream task quality.

Per domain for q3 (position-weighted): code KL 0.0192 / top-1 96.88%; English essay 0.0379 / 93.44%; math 0.0235 / 95.50%; Chinese chat 0.0650 / 89.65% (the weakest domain, below 90% on its own).

The offload runtime itself was checked to be exact: LRU / LRU+prefetch outputs were bit-identical to a read-all-experts reference path (2 prompts × 32 greedy tokens), and the deployed loader matched an independent rebuild of the same weights with KL 0.0.

Speed (128 new tokens, greedy, 3 rounds, valid rounds only, expert cache fixed at 24.9 GB)

• Decode was 2.2–2.6 tok/s across the plain-LRU arm above (2.24–2.47) and a variant with the first MoE layer pinned in memory (2.33–2.56; not exposed in nemo_offload.py). Prefetch on: 1.97–2.07 tok/s (slower, hence off by default).
• Prefill for a ~2k-token document: about 32 tok/s (32.2–32.7 across arms). Short prompts prefill slowly (~1.3–1.7 tok/s) because almost every expert they touch is a disk read.
• Memory in these runs: trunk 7.43 GiB, expert cache 24.85 GiB, peak RSS about 28.4 GiB; disk reads about 230–270 MB per decoded token.

Limitations

• Quality numbers come from 8 already-seen sentences; they say nothing reliable about real tasks. Chinese chat agreement is the lowest (89.65%).
• Slow: ~2–2.6 tok/s decode, and short-prompt prefill is ~1.5 tok/s. Speed depends heavily on SSD speed and free memory.
• Only tested on one machine (M4 Pro, 64 GB). Smaller machines get a smaller expert cache and will be slower; behaviour there is untested.
• Text only; no vision, no MTP speculative decoding. Batch size 1. Long-context behaviour beyond ~2k tokens was not measured.
• nemo_offload.py is the offload path of our internal loader with the measurement code removed; the published file itself has passed static checks but the numbers above were produced by the internal version.
• All the safety, bias and usage notes of the original NVIDIA model card apply.

License

Distributed under the OpenMDW License Agreement, version 1.1, the same license as the original model. A copy is in LICENSE. Original model © NVIDIA Corporation ("NVIDIA Nemotron" is a trademark of NVIDIA). FP8 checkpoint by RedHatAI. This repository is an independent conversion and is not affiliated with or endorsed by NVIDIA or Red Hat.

---

繁體中文

NVIDIA Nemotron 3.5 Super（總參數 120B／啟用 12B，Mamba-Transformer 混合、latent MoE）的純文字 MLX 轉換版，給 Apple Silicon 使用。 路由專家（routed experts）放在磁碟上、需要時才讀取。40 個 MoE 層的路由專家全部是 3-bit；常駐的主幹（trunk）是混合精度（見下）。本檔沒有三值權重。

這不是標準 mlx-lm 格式。 mlx_lm.load() 無法載入（已用 mlx-lm 0.32.2 實測：報 FileNotFoundError: No safetensors found，因為權重放在 trunk/ 與 experts/，不是 model*.safetensors）。 請用本 repo 附的 nemo_offload.py。

出處

• 原模型：nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16（NVIDIA）
• 轉換來源：RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-FP8 @ 38effd54b334bfda8fce082f4290aa56d86b7533（FP8 反量化後重新量化為 MLX affine）
• 授權：沿用原模型的 OpenMDW License Agreement 1.1（全文，副本見 LICENSE）
• 只含文字部分；原模型的視覺編碼器與 MTP（多 token 預測）頭不包含在內。

內容

• experts/experts-LL.safetensors（40 檔）：MoE 第 LL 層的 512 個專家，MLX affine 3-bit、group 64，共 45.94 GiB
• trunk/*.safetensors（90 檔）：embedding、lm_head、Mamba／attention／共享專家／latent 投影／router。embedding、lm_head、latent 投影、共享專家、Mamba in_proj 為 8-bit；Mamba out_proj 與 attention q/k/v/o 為 4-bit（group 32）；norm、router、conv、SSM 參數不量化。共 7.43 GiB
• config.json、tokenizer 檔、chat_template.jinja、generation_config.json、nemo_offload.py、requirements.txt
• 總計 53.39 GiB（57.32 GB）

執行方式

pip install -r requirements.txt
hf download Jakevin/NVIDIA-Nemotron-3.5-Super-120B-A12B-q3-mlx --local-dir ./nemotron-q3
python ./nemotron-q3/nemo_offload.py --model-dir ./nemotron-q3 --prompt "用兩句話解釋混合專家模型。"
python ./nemotron-q3/nemo_offload.py --model-dir ./nemotron-q3 --interactive

參數：--max-tokens、--temp（0＝greedy，預設）、--top-p、--thinking、--expert-cache-gb（預設用記憶體規則允許的全部）、--prefetch（跨層預取，預設關閉；實測較慢）。

實測數字（請先看限制）

以下全部是我們自己在一台 Mac mini（Apple M4 Pro、64 GB 統一記憶體、macOS 26.6.2），專家從外接 USB SSD 讀取上量到的，mlx 0.32.4／mlx-lm 0.32.2。出處：執行紀錄 RESULTS_remedy.md（2026-10-18）。

品質（對 FP8 參考，僅屬工程試驗）：參考＝RedHatAI FP8 版反量化後以 BF16 計算。資料是 8 句開發用短序列（程式、英文短文、數學、中文對話），共 1,457 個位置，teacher-forced；這 8 句在開發過程中已看過、不是 held-out，也沒有信賴區間。這只是工程檢查，不是 benchmark。

q3 逐域：程式 KL 0.0192／96.88%；英文短文 0.0379／93.44%；數學 0.0235／95.50%；中文對話 0.0650／約 89.65%（最弱的一域，單獨看低於 90%）。

速度（生成 128 token、greedy、3 輪只計有效輪、專家快取固定 24.9 GB）：decode 2.2–2.6 tok/s（純 LRU 2.24–2.47；第一個 MoE 層常駐的變體 2.33–2.56，nemo_offload.py 未提供此選項；開預取反而 1.97–2.07）。約 2k token 文件（doc2k，2,107 token）的 prefill 約 32 tok/s；短 prompt 的 prefill 只有約 1.3–1.7 tok/s。執行時主幹 7.43 GiB、專家快取 24.85 GiB、峰值 RSS 約 28.4 GiB。

限制

• 品質數字只來自 8 句已看過的句子，不能代表實際任務表現；中文對話一致率最低（89.65%）。
• 速度慢（decode 約 2–2.6 tok/s），且高度依賴 SSD 速度與可用記憶體。
• 只在一台 M4 Pro 64 GB 上測過；記憶體較小的機器專家快取會變小、會更慢，未實測。
• 只有文字；沒有視覺、沒有 MTP 推測解碼；batch 1；超過約 2k token 的長上下文未量測。
• nemo_offload.py 是內部 loader 的 offload 路徑去掉量測程式後的版本；已過靜態檢查，但上面的數字是用內部版本量的。
• 原 NVIDIA 模型卡中的安全、偏誤與使用說明全部適用。

授權

以原模型相同的 OpenMDW License Agreement 1.1 發佈，副本見 LICENSE。原模型 © NVIDIA Corporation（NVIDIA Nemotron 為 NVIDIA 商標）；FP8 版由 RedHatAI 製作。本 repo 為獨立轉換，與 NVIDIA、Red Hat 無隸屬或背書關係。