Qwen3.8-35B-A3B-Distill GGUF

Community GGUF quantizations of empero-ai/Qwen3.8-35B-A3B-Distill.

☕ If this GGUF made your day easier, a coffee would make mine.
Send a coffee ☕
I build and test these releases myself. Your coffee helps keep me going.
Thank you for supporting this work.

About Qwen3.8-35B-A3B-Distill

This release is based on the upstream Qwen3.8-35B-A3B-Distill model card. The model is a text-generation MoE model with 35B total parameters and approximately 3B active parameters per token, using 256 experts with 8 routed experts. It uses hybrid attention components and has a native context length of 262,144 tokens in the upstream configuration.

The GGUF files are intended for local inference with llama.cpp and compatible runtimes. The upstream model card and runtime support should be consulted before using long-context, tool-calling, or other task-specific features.

No training or fine-tuning was performed for this release. These are community GGUF quantizations, not an official Qwen3.8 release or endorsement by the upstream authors.

Files

Fidelity measurements

The measurements below compare each published GGUF with the locked BF16 GGUF source. They are next-token fidelity measurements, not a general capability benchmark. Values are averaged over WikiText-2 valid and test, using 2 chunks per split, context length 2048, and the same llama.cpp runtime. Lower KLD, ΔPPL, and RMS Δp indicate closer next-token behavior to BF16; higher Top-1 agreement is better.

For a general memory/quality balance, Q4_K_M is the practical starting point from this measurement set. Q3_K_M, Q2_K, and IQ2_XS should be treated as experimental choices where memory limits justify the measured fidelity trade-off. IQ2_XS is the smallest published option in this update and shows a substantially larger fidelity deviation from BF16. Quality summaries and exact reproduction details are in reproducibility/quality-summary.tsv and reproducibility/manifest.md.

Quick start

From a llama.cpp checkout, with reproducibility/chat_template.jinja downloaded from this repository:

./llama-cli \
  -m Qwen3.8-35B-A3B-Distill-Q4_K_M.gguf \
  --chat-template-file reproducibility/chat_template.jinja \
  --jinja \
  --reasoning off \
  --cpu-moe \
  -p 'Answer briefly in English: What is GGUF and why is it useful for running language models locally?' \
  -n 128 -c 4096 -ngl 8

Adjust -ngl for the available GPU memory. The validation profile for this release used 8 GPU layers, CPU MoE, and llama.cpp's qwen35moe support on an NVIDIA A10M.

Reproducibility and validation

• SHA256SUMS.txt contains checksums for the public files.
• reproducibility/manifest.md records the locked BF16 source, calibration data, imatrix, runtime, commands, and validation profile.
• RELEASE_MANIFEST.json records the machine-readable source and artifact ledger.
• reproducibility/quality-summary.tsv contains the machine-readable fidelity table.
• reproducibility/runtime-summary.tsv contains the optional Q4_K_M llama-bench result.
• reproducibility/chat_template.jinja is the upstream chat template used by the smoke test.

Raw conversion, quantization, smoke-test, fidelity, and benchmark logs are kept locally under reports/ and are not part of this public package.

License and attribution

The upstream model is released under the Apache-2.0 license. See LICENSE and the upstream model card for the original model's terms and attribution.

These are community GGUF quantizations, not an official Qwen3.8 release or endorsement.