Qwen3.8 Flash-Next for tqLLM
Run Qwen3.8 Flash-Next (125B mixture of experts) with tqLLM on one 24 GB NVIDIA GPU or a DGX Spark / GB10. Two ways in:
- The public IQ3_S file + this repo's small files. The ISTA-DASLab GSQ-RCO IQ3_S GGUF is the only large download; tqLLM rearranges its experts in memory at start-up.
- tqLLM's own Flash-Next file (
mixed/, 82.4 GB in two parts): the same IQ3_S experts, IQ4_XS dense layers and 8-bit hyper-connection mixers. Faster on both kinds of hardware:
| Measured | tqLLM's own file | Comparison |
|---|---|---|
| DGX Spark, prompt processing 8K / 16K / 32K / 64K | 4.32 / 8.14 / 16.83 / 35.64 s | TensorFold 1.2.2: 4.90 / 9.21 / 18.58 / 38.11 s |
| DGX Spark, decode, one request | 122 tok/s | TensorFold 1.2.2: 73 tok/s |
| RTX 3090, 145K-token prompt: first token / decode | 58.6 s / 76.1 tok/s | IQ3_S file: 61.7 s / 71.6 tok/s |
Same prompts and client for both sides; GSM8K-200 96.0 % for this file. Method and more rows: tqllm.com/performance, release notes.
| File | Size | What it is |
|---|---|---|
mixed/Qwen3.8-Flash-Next-tqLLM-00001-of-00002.gguf, -00002-of-00002.gguf |
44.7 + 37.8 GB | tqLLM's own Flash-Next file (optional) |
flashnext-tqllm.tqfn |
1 KB | the manifest for tqLLM's own file |
flashnext.tqfn |
1 KB | the manifest for the public IQ3_S file |
mtp-blk48-iq4.gguf |
1.46 GB | the model's own multi-token-prediction layer, IQ4, for exact speculative decoding |
flashnext-tokenizer.gguf |
11 MB | tokenizer and chat template |
atx_65536.txt |
0.4 MB | draft vocabulary for the MTP drafter |
Run
Needs a 24 GB NVIDIA GPU (RTX 30 or newer) or a GB10 / DGX Spark, 64 GB of system memory or more (the experts live there; the numbers above used 128 GB) and tqLLM 1.4.2 or newer. With the public IQ3_S file, put everything in one folder:
hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF --include "IQ3_S/*" --local-dir fn
mv fn/IQ3_S/* fn/
hf download YTan2000/Qwen3.8-Flash-Next-tqLLM --exclude "mixed/*" --local-dir fn
docker run --rm --gpus all -p 127.0.0.1:8086:8086 -v "$PWD/fn:/fn:ro" --entrypoint tqserve \
ytan2000/tqllm:1.4.2 --model /fn/flashnext.tqfn --tokenizer /fn/flashnext-tokenizer.gguf \
--context 32768 --host 0.0.0.0 --port 8086
With tqLLM's own file instead (no ISTA download needed):
hf download YTan2000/Qwen3.8-Flash-Next-tqLLM --local-dir fn
docker run --rm --gpus all -p 127.0.0.1:8086:8086 -v "$PWD/fn:/fn:ro" --entrypoint tqserve \
ytan2000/tqllm:1.4.2 --model /fn/flashnext-tqllm.tqfn --tokenizer /fn/flashnext-tokenizer.gguf \
--context 32768 --host 0.0.0.0 --port 8086
Then curl -s http://localhost:8086/health and any OpenAI-compatible client at http://localhost:8086/v1. The server
is ready after about two minutes.
Credits and licence
Model: Qwen team (Qwen/Qwen3.8-Flash-Next), Qwen Community License 1.0
(copy in LICENSE-Qwen; keep it with any copy of these files). Routed experts: IST Austria DAS Lab (GSQ + RCO),
Apache-2.0. Importance matrix for the IQ4_XS dense layers: unsloth. tqLLM's own file and tqLLM: David Y. Tan. See NOTICE.