EXL3 quantization of xtools-at/DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic at revision e4bfa83c72896cbb4990c909967d6f27bb97bac4. Original weights without a LoRA adapter; the upstream tokenizer and chat template are preserved.
| Setting | Value |
|---|---|
| Main linear weights | 8 bits per weight target |
| Output head | 8 bits |
| Vision tower | 6 bits |
| MTP layers | 4 bits |
| Token embeddings | BF16 (16 bits) |
| ExLlamaV3 converter | 1.6.6 |
| Codebook | mul1 |
| Calibration | Bundled corpus, 250 rows × 2048 tokens |
| Output scales | auto |
| Weight files | 10,672,495,208 bytes |
The target bitrate applies to the main linear weights. Embeddings, the head, vision/MTP components, scales and metadata have their own storage, so the total file size is higher than a uniform 8-bit estimate.
Load this folder with ExLlamaV3 1.6.6 or a compatible ExLlamaV3 backend such as TabbyAPI. Use the included original chat template. KV cache precision is a runtime setting; Q8 was used for local text-generation checks. Quantization does not change the architecture's configured context limit. Memory use depends on context length and which components are loaded.
See conversion.json for the source pin, tensor checksums and reproducible converter arguments. smoke-test.json, if present, records short deterministic text-generation checks and their context/cache settings; these are not a coding benchmark or a validation of the full context window, vision tower or MTP.
8 bpw had the lowest KL divergence from the original model; 5 bpw had the highest observed generation throughput. All three passed 5/10 selected hard exercises with no LoRA. The pass totals hide different edge-case failures and do not establish equal general correctness.
| EXL3 | Mean KL, nats ↓ | Reference top-1 agreement | Strict passes | Generation tokens/s | Primary task minutes | Reported completion tokens |
|---|---|---|---|---|---|---|
| 8 bpw | 0.000132 | 99.54% | 5/10 | 34.9 | 45.4 | 76,995 |
| 6 bpw | 0.001128 | 98.38% | 5/10 | 42.4 | 47.7 | 76,971 |
| 5 bpw | 0.003654 | 97.74% | 5/10 | 44.7 | 42.1 | 71,721 |
These timing and token columns cover the first attempts only. Throughput is total reported completion tokens divided by total server generation time. Wall time is the sum of OpenCode task times, excluding setup, grading, KL probes, and extra retries. Cancelled streams can lack final token-usage records. Measurements are single runs on a shared RTX 3060 12 GB desktop.
D_KL(P_reference || P_quantized) was measured in natural-log nats at temperature 1 over the full 248,077-token vocabulary, without top-k truncation. The reference is the original unquantized BF16 checkpoint at the source revision above, loaded into ExLlamaV3's FP16 runtime. Logits were FP16 and distribution reductions FP64. Both reference and quantized models used cache-free teacher-forced forward passes with identical Q8 K/V simulation.
The probe scored 8,192 next-token positions from 16 sequences of 513 input tokens: 4,096 prose positions from WikiText-2 raw test and 4,096 Python positions from HumanEval test prompts plus canonical solutions. Windows were fixed, nonoverlapping within each domain, with seed 3407; the nine coding challenge IDs below were excluded from the code corpus. Dataset revisions, offsets, tokenizer and input hashes are in the KL manifest. Tokenizers were identical across all variants. Reference self-KL was zero, the three reference NLL arrays matched exactly, and saved per-token metrics were checked against the aggregates.
This is a short-context text diagnostic, not a long-context, vision, or MTP evaluation. Dataset test splits do not establish absence from pretraining or calibration. Lower KL measures fidelity to this reference distribution; it does not guarantee better coding accuracy.
The same ten hard replay exercises were attempted using OpenCode v2.2.7 in auto/YOLO mode, with tool permissions allowed: HumanEval 1, 6, 32, 39, 76, 99, 101, 132, 141, plus the TerminalBench pytorch-model-cli exercise. The grader was reconstructed-base-plus-v1, not a certified EvalPlus run. A strict pass requires the base tests and all added cases; function deadlines apply, and the CLI also requires a real ELF executable and functional artifact checks. Final files were graded independently after agent completion, error, or timeout.
All variants used ExLlamaV3 1.6.6, the original chat template with thinking enabled, no LoRA, no Sharp template, no MTP draft, Q8 cache, a 131,072-token server/cache limit, an 80,000-token OpenCode context budget, batch size 1, and chunk size 512. Sampling was temperature 0.6, top-p 0.95, top-k 20, seed 3407, with 8,192 output tokens per response and 20 minutes per exercise. These configured limits do not imply that a full 80K/128K context was exercised.
Failed artifacts with an explicit output-limit error were eligible for a separate 32,768-token-per-response retry at the same 20-minute exercise cap. Only 8 bpw and 5 bpw HumanEval/132 qualified; both completed their valid retries and still failed the strict oracle, leaving all scores at 5/10. Their additional times were 9.2 and 17.3 minutes, respectively. The first extended 5 bpw attempt hit an inherited HTTP timeout; it was excluded and rerun with a corrected transport timeout. Valid retries and excluded harness overhead are identified separately in the results file.
The 8 bpw root-finder returned 0.0 for x^3 - 2x + 2, leaving a residual of 2.0. The 6 bpw root-finder raised an overflow error on a base test, and its filename checker incorrectly accepted example.TXt despite the specified lowercase extensions. All three prime-Fibonacci solutions hit an added-case deadline, and all three CLI attempts reached the 20-minute cap with failing artifacts. These failures leave substantial room for improvement despite the low KL values.
HumanEval/132 has an oracle wording ambiguity. For [][][], the strict oracle expects False, while the completed 8/5 bpw extended responses returned True. The prompt says “subsequence”: selecting indices [0, 2, 3, 5] gives the noncontiguous nested subsequence [[]]. The canonical oracle instead checks nesting inside contiguous balanced groups. Strict scores above are retained without adjustment; this case should not be read as an unambiguous algorithm error.
Download machine-readable results and per-task metrics, comparison CSV, and verified failure notes. This is a ten-task challenge replay, not the full HumanEval/EvalPlus benchmark or a general accuracy estimate. The original model was used as the KL reference but was not given a matching full-precision correctness run in this comparison.
Upstream model: xtools-at/DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic. Upstream ancestry includes Jackrong/DeepSeek-V4-Pro-Qwen3.5-9B. The source model card declares Apache-2.0; that license is included in LICENSE.
The upstream model card is preserved in UPSTREAM_README.md. Its training, distillation and performance statements are upstream claims, not independently established by this conversion.
model-00001-of-00003.safetensors 4.2 GBmodel-00002-of-00003.safetensors 4.1 GBmodel-00003-of-00003.safetensors 2.4 GB