Qwen3.8 Flash-Next MLX SSD Stream

Quality-first Flash-Next at 70+ tok/s on a 128 GB Mac

This prepared model prioritizes quality and speed: a BF16 core and verification path, Q4 routed experts, native MTP, multimodal support, and the full 262,144-token context window. On an M4 Max with 128 GB unified memory, matched warm workloads measured 71-74 tok/s for code and technical prose.

SSD Stream makes that configuration fit without keeping the model's 51.2 GB lookup table in unified memory. The table streams from SSD while the GPU is already working, leaving that memory available for model weights, KV cache, and macOS. A Q6 copy of the vocabulary head proposes MTP tokens that are always checked by the untouched BF16 verification head.

What this model is

The prepared repository contains the complete model and a read-only ngram_table.bin; no conversion step or companion draft model is required. The table payload is source-verified during conversion and remains in its original FP8 E4M3 format.

Performance

M4 Max, 128 GB

The decode comparison used the 262,144-token server context and native MTP at depth 4. Rates are completion tokens divided by total request wall time, median of three warm runs after one discarded warmup. Each pair used the same prompt and greedy sampling.

The uncached 248,445-token retrieval test returned its exact buried key and stopped normally within the native context window. The release also passed 39 end-to-end checks covering text, structured tools, streaming, MTP, long-context retrieval, images, concurrent decode, and mixed speculative and serial traffic.

Run it

Build mlx-serve, then give it this Hugging Face model ID:

git clone --recurse-submodules https://github.com/garnermccloud/mlx-serve
cd mlx-serve
brew bundle install --file=Brewfile
./app/build.sh
./zig-out/bin/mlx-serve run garnermccloud/Qwen3.8-Flash-Next-MLX-SSD-Stream

The first run downloads the prepared files and then starts the server at http://localhost:11234. Later launches reuse the verified local snapshot. Native MTP and SSD streaming are selected from the model metadata, so there are no model-specific serving flags.

Capabilities

• 262,144-token native context
• text, image, and video input
• reasoning and structured tool calls
• OpenAI Chat Completions, Responses, Anthropic Messages, and Ollama APIs
• native MTP enabled automatically; no companion model or serving flags

Precision

How SSD Stream works

Flash-Next needs 16 deterministic lookup rows per token. Their addresses are known before the model reaches the block that consumes them, so mlx-serve starts persistent parallel pread operations while the preceding GPU work is still running. It restores rows in token order, converts FP8 directly to BF16, and waits only when storage takes longer than the overlapping computation.

The 51.2 GB table is mapped read-only. SSD Stream does not create a second private copy or reserve a multi-gigabyte cache; macOS may retain useful pages in its reclaimable filesystem cache.

Hugging Face metadata

Hugging Face's automatic parameter and precision labels describe the packed safetensors, not the complete source architecture. The transformer has 125B parameters with 6B active per token. The architecture also includes the 51.2-billion-element PLE table and MTP weights, bringing its logical total near 180B. The PLE is stored separately in ngram_table.bin, so it is not included in the safetensors parameter count.

License and attribution

The model retains the Qwen Community License. Transformer and vision weights come from Qwen. The FP8 table comes from the RadixArk NVFP4 checkpoint. Serving support and the prepared layout are provided by garnermccloud/mlx-serve, an independent fork of mlx-serve.