The fastest inference engine for Qwen3.8-Flash-Next on Blackwell

Model files for Basalt, a local inference engine for the Qwen3.8-Flash-Next mixture-of-experts model. Basalt runs on Linux with any NVIDIA Blackwell GPU (sm_120). A second GPU can serve as a peer expert tier, and experts that do not fit in VRAM stay in system RAM. It has an OpenAI- and Anthropic-compatible API.

Each .basalt file is self-contained: weights, tokenizer, MTP drafter, vision projector and the int8 hyper-connection table the engine reads at start. The expert profile (which experts start in VRAM) ships with the engine. Expert formats: GSQ-RCO (IQ3_XXS, IQ3_S, Q2_0), Unsloth UD K-quants (Q4_K_XL, Q5_K_XL, Q6_K_XL), Q8_0 and NVIDIA NVFP4.

Files: Qwen3.8-FN-NVFP4-DQ6_K, -NVFP4-DQ8, -UD-Q4_K_XL, -UD-Q5_K_XL, -UD-Q6_K_XL, -Q8_0, -GSQ-RCO-IQ3_S, -GSQ-RCO-IQ3_XXS and -GSQ-RCO-Q2_0 (each .basalt), plus convert/.

These are .basalt format version 2 files, which Basalt 0.2.0 and later require. Version 2 drops engine policy data from the pack (the expert profile now ships with the engine, so a better one never needs a new download) and stores the int8 hyper-connection table, so the engine no longer rebuilds it at every start. Weights are unchanged and output is identical. Already have a version 1 file? Re-downloading is optional: python3 tools/migrations/basalt_v1_to_v2.py OLD.basalt NEW.basalt converts it locally without re-quantizing (needs numpy and room for a second copy). See the file format changelog.

Speed

Best measured, single stream, 400 W, RTX 5090 + RTX 5060 Ti peer, tokens/s. Numbers move with every release.

Quality

Measured with basalt-bench --quality: 100 chunks of wikitext-2 (512 tokens each), scored through the decode path against full precision: a llama.cpp BF16 reference of the original model (reference PPL 3.273 on the scored rows). int8 KV, RTX 5090 at 400 W + RTX 5060 Ti at 180 W. Lower KLD and higher top-1 agreement are closer to full precision. PPL is the pack's own perplexity on the same text; it is not a closeness measure. Only the official Basalt packs are listed.

Both NVFP4 packs keep NVIDIA's NVFP4 experts bit for bit; the head and embedding are Q8_0. The D suffix names the dense projections' format: Qwen3.8-FN-NVFP4-DQ6_K stores them as Q6_K, quantized once from BF16, and Qwen3.8-FN-NVFP4-DQ8 stores them as Q8_0, which removes most of the 6-bit pack's measurable rounding (KLD 0.0059 lower; every rounding removed, BF16 dense included, is 0.003 lower still) and decodes about 7% slower on prose and 3% on structured output. Every rounding the engine does is measured in the precision docs.

The IQ3_S, IQ3_XXS, Q2_0 and NVFP4 packs carry a self-distilled MTP layer (convert/mtp/rt/dense.bin) for higher draft acceptance; the others keep the original.

Run

Build the engine as described in the Basalt README, then:

hf download jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-NVFP4-DQ6_K.basalt --local-dir models

basalt_config.json:

{
  "server": { "model": "models/Qwen3.8-FN-NVFP4-DQ6_K.basalt", "gpu": "0,1", "port": 8080 },
  "engine": { "expert-cache": "auto", "prefill": "auto", "kv": "int8", "resident-experts": true }
}

bin/basalt serve --config basalt_config.json

Conversion kit

convert/ holds the inputs for tools/convert/convert.py (MTP layer, vision projector, draft vocabulary), so any GGUF quant of this model can become a .basalt file. See docs/model-file.md.

Sources

• GSQ-RCO: ISTA-DASLab, IST Austria; also the vision projector.
• UD quants, Q8_0: unsloth.
• NVFP4 experts: NVIDIA.
• MTP drafter: Qwen/Qwen3.8-Flash-Next BF16 MTP tensors.
• Basalt is a fork of Strata (MIT).

Citation

If you use the GSQ-RCO files, please cite:

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026}
}
@article{rco2026,
  title  = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
  author = {Helcig, Michael and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2605.00649},
  year   = {2026}
}

License

All files here are derived from Qwen3.8-Flash-Next and are distributed under the Qwen Community License 1.0, whose notice is included in this repository and inside every .basalt file. Businesses offering Model as a Service or an AI coding/office assistant need a separate license from Qwen for commercial use, and products above 100M monthly users or US$20M monthly revenue must display the model name. The LICENSE has the binding terms. The Basalt engine itself is MIT.