Run JANG models in MLX Studio / vMLX

⚠️ Runtime: vMLX (Python) support for this bundle is not in a released build yet. It was built and validated on a vMLX development build that runs qwen4_exp JANGH bundles with 4- and 6-bit expert widths. Released vMLX builds cannot load it.

JANGQ-AI/Qwen3.8-Flash-Next-JANGH4

Qwen3.8-Flash-Next at near-reference quality for 128 GB Macs. About 74 GiB in memory; the 53.6 GiB n-gram embedding table is read from the SSD on demand. Every reasoning effort closes as often as our 6-bit release, MTP is as good as the bf16 MTP, and image and video input are calibrated.

A JANGH bundle of Qwen/Qwen3.8-Flash-Next: a 125B-parameter MoE (512 routed experts, top-10 + a shared expert), gated-delta-net and sparse-attention layers, one MTP layer, image + video input and a 51B-entry n-gram embedding table.

Read this first: what "agreement" can reach

Scores are against the full bf16 model, run layer by layer from disk. Some next tokens are nearly tied, so any change to the weights flips a few of them. We measured that on the unquantized model: relative noise of 0.001 on a single tensor. That run is the ceiling in the tables below; the distance to the ceiling, not to 100%, is what the quantization costs.

Fidelity vs the bf16 model

510,954 teacher-forced positions on three held-out sets never used for calibration. Top-128 renormalized KL, scored on the bundle exactly as stored.

General set by domain, median KL · top-1 agreement:

Decisions: does it choose like the bf16 model?

Reasoning efforts (served, same 272 prompts as our 6-bit JANG_6S release)

Math, office/finance, networking, science, cybersecurity, coding and agentic prompts, each at every effort.

The efforts behave like the reference: xhigh spends far more tokens on hard prompts (p90 12,928 tokens vs 2,912 at low; JANG_6S: 11,386 vs 3,165), and those long traces close at the reference's rate.

MTP (speculative decoding)

The MTP proposal head ships as a calibrated 4-bit refit of this bundle's 8-bit lm_head (mtp_draft/, described by vmlx_mtp_proposal_head.json at the bundle root).

Image and video (served)

Speed and memory (vMLX Python, M5 Max 128 GB, A/B/A against JANG_4M)

Same runtime build, one model loaded at a time, order JANGH4 → JANG_4M → JANGH4; each value is the median of the warm runs (the first, cold run of each set is dropped). MTP off. Decode is on par with the plain affine 4-bit quant (the 8-bit attention is the largest share of the bytes read per token); long-prompt prefill is 14-37% faster. ¹ Run A′'s two warm prefill runs were 1,595 and 1,112 tok/s.

JANGH

JANGH is our expert format: a fixed blockwise Hadamard-32 transform with no random signs and a closed-form codebook per bit width (odd cubic at 2-4 bits, linear at 6 and 8), with one fp16 scale per row, stored bit-for-bit in MLX's own packing layout so a runtime reuses MLX's kernel structure for decode and prefill. Rounding is GPTQ on each expert's own input covariance.

For compatibility with runtimes already built against it, the on-disk identifiers keep their original names: config.json → jangtq block (version: 2), per-module "mode": "jangtq2", tensors *.tq2_packed / *.tq2_scales, jang_config.format = "jangtq2". They are not loadable by, and must not be routed to, JANGTQ v1 loaders.

What's in the bundle

• Vision + video: Qwen3-VL tower, image and video processor configs.
• MTP: 1 layer plus the calibrated proposal head.
• Thinking + agentic:
  • Thinking is ON by default (the template opens <think>); reasoning is kept in history.
  • Reasoning efforts low / medium / xhigh (default xhigh) via the reasoning_effort chat-template kwarg.
  • Thinking off (enable_thinking: false) renders <think>\n\n</think>\n\n.
• Tool calls: XML function calls (<tool_call> / <function=name> / <parameter=arg>…</parameter>), declared as tool_parser: qwen; reasoning parser qwen3.
• Expert widths differ per projection inside a layer: gate/up 4-bit in all 48 layers; down 6-bit in layers 5, 7-33, 35, 36, 38, 40-46 and 4-bit in the other 10. Each width is declared per module in config.json (quantization) and in jang_config.expert_bits; codebooks for 2/3/4/6 are declared in config.json → jangtq.
• Self-describing: per-module quantization map, the JANGH format block, capabilities, chat / effort / sampling / MTP / context stamps, 128 explicit n-gram table shard rules. Raw evaluation results are in evaluation/.
• Every shard is alignment-safe (zero-copy memory mapping). 65 shards, 3,072 tensors, 125.6 GiB on disk.

Serving contract

• Sampling, thinking: temperature=1.0, top_p=0.95, top_k=20; thinking off: temperature=0.7, top_p=0.8, top_k=20, presence_penalty=1.5. No repetition penalty.
• EOS: [248046, 248044] · context: 262,144 native (YaRN-extensible to 1M)
• Memory: ~74 GiB of weights stay in memory (wire that much, not the folder size); the n-gram table is read from the SSD and is never loaded into memory.
• Continuous batching on the development build: keep at most 6 concurrent sequences (at 7+ the routed-expert path switches to its long-prompt kernels and slows down).

Build details

• Source: Qwen/Qwen3.8-Flash-Next (bf16)
• Calibration: 604,179 tokens rendered with the model's own chat template: images and video (100k, 48 sequences), reasoning traces at every effort (82k), cybersecurity (78k), coding (76k), agentic and tool conversations (75k), terminal (70k), long context (37k), math (25k), science (25k), networking (17k), office and finance (12k), creative (7k). The held-out sets are disjoint from calibration.
• Experts: JANGH, GPTQ on per-expert centered covariances with act-order, all 48 MoE layers; widths placed per projection by measured held-out error per byte; unit-gain row scales (A/B'd against plain GPTQ scales; the difference was within noise)
• Everything else: importance-matrix weighted affine fit + AWQ from the same capture; vision tower from a dedicated image/video capture; MTP from its own capture

Quantized and validated by Jinho Jang — eric@jangq.ai