CyberTiel-Coder-35B-MLX (FP16 tensors · fixed MTP head · tuned for oMLX on Apple Silicon)

English · 中文

A 35B-A3B MoE coding/agent model (6-bit oQ6e, vision-capable, abliterated) in the exact form we run every day on a Mac Studio M1 Ultra 64 GB with oMLX 0.7.0. Compared with the upstream pack it has two local changes, both measured:

1. Non-quantized BF16 tensors cast to FP16 (M1/M2 have no native bfloat16). Prefill +28–50 %, decode at 16K context +22 %, quality unchanged (206/260 = 206/260).
2. Rebuilt MTP head ("fixedmtp": 42-key fused layout) so that oMLX's Lightning MTP speculative decoding runs with an adaptive depth → 94–116 tok/s chat decode.

Everything below is measured on one machine; numbers are not a promise for yours. Full method and raw data: tuning notes dataset · GitHub.

⚠️ Abliterated (refusal-removed) model. Meant for emergency / red-team / research work. Run it inside an OS-level sandbox with restricted network and execution rights, and guard against prompt injection. You are responsible for how you use it.

Updated 2026-10-10: the retrieval pair of this stack is now BGE-M3 8-bit + bge-reranker-v2-m3 (FP16 embedding tables); the previous embedding model, Qwen3-Embedding-0.6B, was retired. 更新于 2026-10-10：本机栈的检索组合现为上述 BGE-M3 8-bit + bge-reranker-v2-m3，旧嵌入模型 Qwen3-Embedding-0.6B 已下线。

At a glance

Lineage: ornith-ai/Ornith-1.5-35B-A3B → huihui-ai/Huihui-Ornith-…-abliterated (refusal removal) → peculiar-ragdoll/Cyber-Tiel-Coder-35B-A3B-MLX-oQ6e-MTP (rev a443d4e) → our fixedmtp head → FP16 tensor cast (this repo). The previous BF16 build of this repo stays reachable as revision 32e067dd15c07fbfcf841d53efd011f90c6527b7 (recommended for Macs with native bf16, M3 and later — untested by us).

Measured results

1. BF16 → FP16 on M1 (the biggest free win)

What was cast: 1569 tensors (scales, biases, norms, embeddings, MTP head, vision tower; 2.94 GB). Max |x| = 25.4 (far inside the FP16 range), 26 103 tiny values underflowed to zero. The quantized U32 data and model.safetensors.index.json are unchanged. Script: recipe/to_fp16_v2.py (aborts if any value would overflow). A pure short-decode test showed no gain and misled us once — always measure prefill at 1K/4K/16K and 16K decode.

2. MTP depth (Lightning MTP, chat decode tok/s, BF16 baseline sweep)

Leave mtp_fixed_depth and mtp_adaptive_max_depth empty (0.7.0 model default = adaptive): 94.2 tok/s vs 89.8 for fixed d2, 84.7 for d3, 59.4 with MTP off. Note: oMLX 0.7.0 silently ignores the old mtp_num_draft_tokens field.

3. Long context (hot cache off)

Peak memory with this model alone at 128K: 36.8 GB (fits the default 48 GB Metal limit on a 64 GB Mac).

4. Code quality depends on the thinking mode you use

We rejected a newer upstream requant (09-29) and a rival code model after testing in the thinking-on production mode (LiveCodeBench-60 subset 38 vs 40; 324 code questions 247 vs 242, p = 0.30). Benchmarking with thinking off would have picked the wrong model.

5. What we tried and rejected for this model

TurboQuant KV (decode −20 %), ANE prefill (slower), DFlash / SpecPrefill (no gain), model_type_override=llm to save memory (decode −23 %, image requests silently hallucinate — never do this), hot cache on 64 GB (starves long-context admission).

Recommended oMLX settings

Per-model entry: omlx/model_settings.entry.json (key = model directory name).

{
  "max_context_window": 131072, "max_tokens": 32768,
  "temperature": 0.6, "top_p": 0.95, "top_k": 20, "repetition_penalty": 1.0,
  "enable_thinking": true, "thinking_budget_enabled": false,
  "mtp_enabled": true, "vlm_mtp_enabled": false,
  "ttl_seconds": 1800, "model_alias": "CyberTiel-35B"
}

Global (see the dataset repo's config/ for the full redacted files): hot cache off (hot_cache_max_size: "0"), SSD cache on, max_concurrent_requests: 8, memory_guard_tier: balanced with soft_threshold left at 0.85 (0.70 legacy values make models evict each other), TurboQuant/DFlash/SpecPrefill/ANE prefill all off.

Thinking control

Quick start

# 1) download (mirror example for mainland China)
HF_ENDPOINT=https://hf-mirror.com hf download YCF-AI/CyberTiel-Coder-35B-MLX \
  --local-dir ~/models/CyberTiel-Coder-35B-MLX
# 2) point oMLX model_dirs at ~/models, merge omlx/model_settings.entry.json into ~/.omlx/model_settings.json, restart oMLX
# 3) chat
curl http://127.0.0.1:8000/v1/chat/completions -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"CyberTiel-35B","messages":[{"role":"user","content":"Write a Python LRU cache."}],"reasoning_effort":"low"}'

The MTP head (model-mtp.safetensors) is used by oMLX's Lightning MTP; we have not tested other runtimes with it. Verify downloaded files with the SHA-256 values in the HF file listing.

Moving to another machine — checklist

1. Install the same oMLX version (0.7.0 or newer), put the model folder under a model_dirs path.
2. Merge the settings entry above; keep MTP depth fields empty.
3. Disable hot cache, keep concurrency 8, keep the memory guard at balanced.
4. Measure your machine: 3-prompt chat decode, prefill 1K/4K/16K, 16K decode, then (only if you changed something) a fixed quality set with McNemar. Our protocol is in the dataset repo.
5. If the chip has native bf16 (M3+), compare the BF16 revision with this one before choosing.

Known limits

• Vision is an emergency capability: digit OCR errors, weak counting. Use a dedicated VLM for precision (MiniCPM-V-4.6 build).
• Abliteration removes refusals, not hallucinations.
• All speed numbers: Mac Studio M1 Ultra 64-core GPU, 64 GB, macOS 15.8, oMLX 0.7.0 (build 2987), MLX 0.32.2, mlx-lm 0.31.4.dev, mlx-vlm 0.7.1.

---

中文说明

English · 中文

35B-A3B MoE 代码/智能体模型（6-bit oQ6e、带视觉、去审查），就是我们每天在 Mac Studio M1 Ultra 64 GB 上用 oMLX 0.7.0 实际运行的版本。相对上游打包有两处本地改动，均有实测：

1. 非量化 BF16 张量转 FP16（M1/M2 没有原生 bf16）：prefill +28–50 %、16K 上下文解码 +22 %，质量不变（260 题 206 = 206）。
2. 重建 MTP 头（fixedmtp，42 键融合布局），配合 oMLX Lightning MTP 自适应深度，对话解码 94–116 tok/s。

所有数字来自同一台机器，不保证在你的机器上一致。完整方法与原始数据见调优笔记数据集、GitHub。

⚠️ 去审查（abliterated）模型，仅用于应急测试 / 红队 / 研究。请放进 OS 级沙箱，限制网络与执行权限，并防范提示注入；使用责任自负。

概览

血统： ornith-ai/Ornith-1.5-35B-A3B → huihui-ai/Huihui-Ornith-…-abliterated (refusal removal) → peculiar-ragdoll/Cyber-Tiel-Coder-35B-A3B-MLX-oQ6e-MTP（rev a443d4e）→ 自建 fixedmtp 头 → FP16 张量转换（本仓库）。本仓库此前的 BF16 版仍可用 revision 32e067dd15c07fbfcf841d53efd011f90c6527b7 取回（原生支持 bf16 的 M3 及以后机型可优先对比，我们未测）。

实测结果

1. BF16 → FP16（M1 上最大的免费收益）——图见上（英文部分 fp16 cast）。同时段 ABBA：prefill 1K/4K/16K 1085/1406/1096 → 1393/1891/1644 tok/s；16K 首字 14.9 → 10.0 s；16K 上下文解码 96 → 117；对话解码 102.5/108.7 → 109.1/112.3；260 题 206 = 206。 转换了 1569 个张量（scales、biases、norm、embedding、MTP 头、视觉塔；2.94 GB），最大绝对值 25.4，26103 个极小值下溢为 0；量化 U32 数据与 index 不变。脚本 recipe/to_fp16_v2.py（任何值溢出即中止）。只测短上下文纯解码会得出"无收益"的错误结论，必须测 1K/4K/16K prefill 与 16K 解码。

2. MTP 深度——mtp_fixed_depth 与 mtp_adaptive_max_depth 留空（0.7.0 默认=自适应）最优：94.2 tok/s，固定 d2 89.8、d3 84.7、关 MTP 59.4。注意 0.7.0 已静默忽略旧字段 mtp_num_draft_tokens。

3. 长上下文（hot cache 关闭）：16K 1620/~100、32K 1320/86–89、64K 957/71、128K 614（首字 3.6 分钟）/60（prefill/解码 tok/s）。单模型 128K 峰值内存 36.8 GB，64 GB Mac 默认 48 GB Metal 上限内可用。

4. 代码质量取决于你实际使用的思考模式：我们用开思考的生产口径否决了上游 09-29 重量化版与另一款代码模型（LCB60 38 vs 40；324 题 247 vs 242，p=0.30）；关思考评测会选错模型。

5. 试过并否决：TurboQuant KV（解码 −20 %）、ANE prefill（更慢）、DFlash/SpecPrefill（无收益）、model_type_override=llm 省内存（解码 −23 %、图片请求静默幻觉，切勿使用）、64 GB 上开 hot cache（挤占长上下文准入）。

推荐 oMLX 设置

单模型条目见 omlx/model_settings.entry.json（键名=模型目录名），参数见上方英文部分 JSON。全局：hot cache 关（hot_cache_max_size: "0"）、SSD 缓存开、max_concurrent_requests: 8、memory_guard_tier: balanced 且 soft_threshold 保持 0.85（遗留的 0.70 会让模型互相驱逐）、TurboQuant/DFlash/SpecPrefill/ANE prefill 全关。完整脱敏配置见数据集仓库 config/。

思考控制

快速开始

# 下载（国内用镜像、直连不走代理）
HF_ENDPOINT=https://hf-mirror.com hf download YCF-AI/CyberTiel-Coder-35B-MLX --local-dir ~/models/CyberTiel-Coder-35B-MLX
# 把 ~/models 加入 oMLX 的 model_dirs，合并 omlx/model_settings.entry.json 到 ~/.omlx/model_settings.json，重启 oMLX

MTP 头 model-mtp.safetensors 由 oMLX Lightning MTP 使用，其他运行时我们未测。下载后请用 HF 文件列表中的 SHA-256 校验。

换机器部署清单

1. 安装同版本 oMLX（0.7.0 或更新），把模型目录放进 model_dirs。
2. 合并上面的设置条目，MTP 深度字段留空。
3. 关 hot cache、并发 8、内存守卫 balanced。
4. 在你的机器上实测：3 提示词对话解码、1K/4K/16K prefill、16K 解码；若改了东西，再用固定题集 + McNemar 检验。协议见数据集仓库。
5. 芯片支持原生 bf16（M3+）时，先对比 BF16 版本再选。

已知限制

• 视觉仅应急：数字 OCR 会错、计数弱；精确任务请用专用 VLM（MiniCPM-V-4.6 版）。
• 去审查只去掉拒答，不去掉幻觉。
• 全部速度数据：Mac Studio M1 Ultra 64 核 GPU、64 GB、macOS 15.8、oMLX 0.7.0（build 2987）、MLX 0.32.2、mlx-lm 0.31.4.dev、mlx-vlm 0.7.1。