ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram
Weights from bottlecapai/ThinkingCap-Qwen3.8-Flash-Next, quantized using the recipe from Intel/Qwen3.8-Flash-Next-W4A16-AutoRound. Intel supplies the recipe, not replacement weights. ThinkingCap's tokenizer, chat template and vision processor are retained.
Only intended precision deviation from Intel: ngram lookup weights use scaled FP8 E4M3FN instead of BF16. All other precision settings and exclusions are unchanged.
Layer precision
| Component | Checkpoint precision / treatment |
|---|---|
Main routed MoE experts: gate_proj, up_proj, down_proj |
Symmetric INT4, group size 128; W4A16 |
| Ngram lookup tables | FP8 E4M3FN; shared FP32 scale per logical table across splits |
| Other PLE weights and projections | Original 16-bit precision; BF16 source |
Self/linear attention, in_proj_a, in_proj_b, indexers |
Original 16-bit precision; BF16 source |
Routers (mlp.gate), shared experts and their gates |
Original 16-bit precision; BF16 source |
| Hyper-connection weights | Original 16-bit precision; BF16 source |
| Vision encoder and merger | Original precision; excluded |
| Token embeddings and LM head | Original precision; excluded |
| Entire MTP module, including experts and attention | Original precision; excluded |
| Norms, hash/index buffers, other non-target tensors | Original dtype; integer buffers remain integer |
W4A16 means 4-bit weights and 16-bit activations for INT4 targets. No FP8 activation conversion or FP8 KV-cache requirement is introduced; runtime compute/cache dtypes depend on the inference engine.
Quantization recipe
Uses Intel AutoRound, integrated into Intel Neural Compressor.
| Setting | Value |
|---|---|
| INT4 recipe | W4A16, symmetric, group size 128, 200 tuning iterations |
| Calibration | Batch 8, sequence length 2048, 128 samples, seed 42; default dataset |
| Device / export | auto / auto_round (auto_round:auto_gptq packing) |
| Implementation | AutoRound 0.16.1 for compatibility fixes; Intel used 0.15.0 |
Same configured INT4 recipe, not a claim of bit-identical weights. Intel's exclusion list is retained:
lm_head,embed_tokens,visual,linear_attn,self_attn,hyper_connection,mlp.gate,shared_expert,in_proj_a,in_proj_b,ple,mtp,indexer
After INT4 export, *.ngram_embedding[.shard_N].weight is converted using
FP8(clamp(weight / scale, -448, 448)), where scale = amax / 448 (1 for zero
tables). Splits share an FP32 *.ngram_embedding.weight_scale;
text_config.ple_embedding_dtype is set to float8_e4m3fn.
Provenance
- ThinkingCap source revision:
d90c9fedadaf7556df4206e6bea5a6b391b1a58e. - Intel recipe revision:
4c67bf686b7f7fd386bae6b07ab59e8ff1d5b897. - Export/conversion details and package versions:
quantization-run.json. - Attribution does not imply endorsement by Intel, bottlecapai or Qwen.
Inference and limitations
Requires AutoRound INT4 and FP8 PLE support, including shared scales and split tables; vLLM 0.32.0 implements these loading paths. Compatibility is not universal. No benchmarks are claimed; Intel's scores do not describe this ThinkingCap variant. Validate quality, multimodal behavior and serving compatibility. Source-model limitations and safety considerations still apply.
License
Qwen Community License 1.0, unchanged from ThinkingCap; the original LICENSE is retained, not replaced by the quantizer's software license. Commercial conditions include separate licensing for certain Model as a Service and AI Work Assistant businesses. Read the full license; this is not legal advice.