Generated directly from the original merged BF16 checkpoint at revision
2fbed4d2a8cbf450797dfdcd65bb5ccef74569df, not copied from another quantized repo.
The redundant key wrappers were canonicalized without changing tensor payload
bytes. llama.cpp commit c406f4367e19bea7930e7f74e1984c9acc6d44bf produced the
BF16 intermediate, then independently quantized Q6_K and Q4_K_M from it.
| File | Size (decimal GB) | GPU memory observed at 32K context |
|---|---|---|
| Q6_K | 22.43 | about 23.2 GiB |
| Q4_K_M | 17.01 | about 18.4 GiB |
Both retain all 770 GGUF tensors, including the 15 embedded MTP tensors. Tensor names and shapes were checked against the BF16 GGUF intermediate. Checksums and source provenance are included. The separate BF16 vision projector and original VLM configuration are retained; text inference is the validated workload.
llama-server -m Qwen3.8-27B-abliterated-cyber-preview-MTP-Q6_K.gguf \
-ngl 999 -c 32768 -np 1 -fa on --jinja \
--spec-type draft-mtp --spec-draft-n-max 1
For vLLM, use the tested W8A16 companion.
Direct GGUF/vLLM was not accepted: the tested GGUF plugin lacked mappings for
the fused attn_qkv_gate tensor emitted by this llama.cpp conversion. The
llama.cpp speeds below must not be presented as vLLM speeds.
| Variant / engine | Prompt tokens | Output tokens | End-to-end tok/s | Observed stream tok/s |
|---|---|---|---|---|
| Q6_K / llama.cpp MTP=1 | 4832 | 256 | 58.3 | 60.2 |
| Q6_K / llama.cpp MTP=1 | 32 | 256 | 60.4 | 62.2 |
| Q4_K_M / llama.cpp MTP=1 | 4832 | 256 | 66.1 | 68.4 |
| Q4_K_M / llama.cpp MTP=1 | 32 | 256 | 68.5 | 71.4 |
Measured on one NVIDIA A100-SXM4-40GB with a 32,768-token serving context, one sequence, temperature 0 and thinking disabled per request. Tables show medians excluding retained warmups. Stream rates are observed from SSE events, not isolated GPU kernel measurements. Cached and cold TTFT differ; raw records retain that distinction. Six functional checks cover arithmetic, sorting, Chinese translation, executing a small generated Python function, tool-call generation and retrieval. This is not a comprehensive accuracy benchmark. Quantization is lossy. No RL or further fine-tuning was performed. Vision inference was not tested.
The source's native context remains 262,144 tokens. The 32,768-token runtime setting does not extend RoPE or imply that the entire native context was tested on this 40GB card. All upstream credits and Apache-2.0 terms remain applicable.
hotdogs/Qwen3.8-27B-abliterated-cyber-preview