Qwen3.8-27B Cyber Preview — W8A16
A self-produced INT8 weight-only compressed-tensors checkpoint, distinct
from Q6_K/Q4_K_M GGUF. Weights total about 31.62 GB.
Source revision: 2fbed4d2a8cbf450797dfdcd65bb5ccef74569df.
Created with llmcompressor 0.15.0 and compressed-tensors 0.20.1 using data-free
RTN, symmetric groups of 128, and 16-bit activations. Vision weights, embeddings,
lm_head, MTP tensors and small GDN control projections were kept unquantized.
See quantization-provenance.json for the complete exclusions and verified source.
Tested vLLM deployment
vLLM 0.34.0 / PyTorch 2.14.0+cu130, BF16 activations, A100 40GB. This native checkpoint does not require the experimental GGUF plugin. Both ordinary serving and MTP=1 passed the functional checks; MTP=1 was selected for the deployed API. Observed GPU memory was about 34.4 GiB at the tested 32K context.
vllm serve xenoautomaton/Qwen3.8-27B-abliterated-cyber-preview-W8A16 \
--host 127.0.0.1 --dtype bfloat16 --max-model-len 32768 --max-num-seqs 1 \
--max-num-batched-tokens 4096 --gpu-memory-utilization 0.90 \
--language-model-only --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
Private downloads require your own HF authentication. Use a separate inference API key when exposing the service. The deployed public service also requires that key on non-OpenAI inference/control routes.
| Variant / engine | Prompt tokens | Output tokens | End-to-end tok/s | Observed stream tok/s |
|---|---|---|---|---|
| W8A16 / vLLM 0.34.0 MTP=0 | 4832 | 256 | 39.8 | 41.5 |
| W8A16 / vLLM 0.34.0 MTP=0 | 32 | 256 | 41.0 | 41.9 |
| W8A16-MTP / vLLM 0.34.0 MTP=1 | 4832 | 256 | 56.4 | 60.0 |
| W8A16-MTP / vLLM 0.34.0 MTP=1 | 32 | 256 | 66.0 | 68.8 |
Validation scope
Measured on one NVIDIA A100-SXM4-40GB with a 32,768-token serving context, one sequence, temperature 0 and thinking disabled per request. Tables show medians excluding retained warmups. Stream rates are observed from SSE events, not isolated GPU kernel measurements. Cached and cold TTFT differ; raw records retain that distinction. Six functional checks cover arithmetic, sorting, Chinese translation, executing a small generated Python function, tool-call generation and retrieval. This is not a comprehensive accuracy benchmark. Quantization is lossy. No RL or further fine-tuning was performed. Vision inference was not tested.
The source's native context remains 262,144 tokens. The 32,768-token runtime setting does not extend RoPE or imply that the entire native context was tested on this 40GB card. All upstream credits and Apache-2.0 terms remain applicable.
One-cell Colab recovery
Download restore_colab.ipynb and open it in an A100 40GB Colab runtime. Run the single code cell to restore the pinned runtime, download these published weights, and start authenticated vLLM + MTP through Cloudflare Tunnel. No requantization is needed. See recovery details.
The current launcher automatically fits the context to GPU memory at 95% utilization. On the tested A100 40GB it fits 99008 total tokens; an actual 98376-token retrieval probe passed. The native 262144 setting failed the KV-memory check. Earlier 32K performance measurements above remain historical. Enlarging the limit retained about 69 tokens/s on the short-input benchmark and 60 tokens/s on the 4.8K-input benchmark. These are limited functional/performance tests, not comprehensive quality evaluation.