Qwen3.6-27B Unsloth NVFP4 GGUF

[!WARNING] Experimental community conversion

This repository contains an experimental GGUF conversion of unsloth/Qwen3.6-27B-NVFP4.

Upstream llama.cpp added compressed-tensors NVFP4 conversion support in PR #21095. However, this source checkpoint uses a mixed-precision configuration containing multiple quantization groups, including NVFP4-packed weights, FP8 tensors, and unquantized MTP tensors.

This specific GGUF was produced with a local experimental extension for that mixed layout. It has been tested only with the hardware and llama.cpp builds listed below. Compatibility, numerical behavior, and performance may differ across converter revisions, backends, devices, and future llama.cpp versions. Verify output quality before relying on it.

Current local testing does not show a clear generation-speed advantage over Q4_K_M. Treat this repository primarily as an experimental compatibility and research conversion rather than a performance-oriented replacement for mature K-quant GGUF files.

This is a community GGUF conversion for use with llama.cpp.

It is not a fully NVFP4 model. The resulting GGUF uses a mixed layout:

• compatible MLP tensors are preserved as native GGUF NVFP4
• source FP8 tensors are dequantized and stored as Q8_0
• embedded MTP tensors remain in higher precision
• multimodal and vision tensors are excluded from the text-model GGUF

Model Details

• Source checkpoint: unsloth/Qwen3.6-27B-NVFP4
• Base model: Qwen/Qwen3.6-27B
• Format: GGUF
• Architecture: Qwen3.6 27B dense model
• Quantization layout: experimental mixed NVFP4 + Q8_0 + BF16
• Runtime: llama.cpp
• Purpose: experimental local inference, mixed NVFP4 GGUF compatibility, Blackwell kernel testing, and Qwen3.6 MTP speculative decoding

The source checkpoint uses mixed-precision compressed-tensors groups:

• NVFP4-packed MLP weights
• FP8 attention, linear-attention projections, selected MLP tensors, and lm_head
• unquantized MTP tensors

For the GGUF uploaded to this repository, the historical local converter extension handled the stored layout as follows:

• compatible NVFP4-packed tensors were repacked into GGUF NVFP4 blocks
• supported FP8 tensors were dequantized with their scales and converted to Q8_0 with --fp8-as-q8
• tensors without compressed scales were exported using their source floating-point precision

Upstream llama.cpp Support

Two separate llama.cpp changes are relevant:

• PR #21095 adds compressed-tensors NVFP4 conversion support. It normalizes compressed-tensors metadata and scale conventions into llama.cpp's existing NVFP4 conversion path.
• PR #22196 adds native Blackwell NVFP4 CUDA execution support for applicable matrix operations.

These changes solve different parts of the pipeline:

compressed-tensors NVFP4 checkpoint
                |
                | PR #21095: conversion support
                v
             NVFP4 GGUF
                |
                | PR #22196: Blackwell CUDA kernels
                v
            llama.cpp runtime

Support for loading or executing NVFP4 does not guarantee that every operator, tensor shape, or token-generation path uses the fastest available FP4 Tensor Core kernel. End-to-end performance still depends on tensor type, operator shape, kernel dispatch, batch size, device placement, and the llama.cpp revision.

Current Status

The model loads and runs with a recent native sm_120a llama.cpp CUDA build.

On the tested RTX 5070 Ti + RTX 5060 Ti dual-GPU system, the conversion is functional and performs within a few percent of the tested NVIDIA Qwen3.6-27B-NVFP4-GGUF.

The current mixed NVFP4 GGUF does not provide a clear generation-speed advantage over the mature Q4_K_M path.

Latest mixed-task results:

• NVIDIA NVFP4 was about 3% faster than this experimental conversion
• Q4_K_M was about 5% faster than NVIDIA NVFP4
• Q4_K_M was about 8% faster than this experimental conversion in average generation throughput

This conversion is currently most useful for:

• NVFP4 compatibility testing
• Blackwell CUDA-kernel experimentation
• mixed compressed-tensors conversion research
• future llama.cpp NVFP4 regression and improvement comparisons

Important Limitations

This conversion is experimental and is not an official Unsloth, Qwen, NVIDIA, or llama.cpp release.

Known limitations:

• the uploaded GGUF was created using a historical local extension targeting the specific mixed FP8 + NVFP4 layout used by this checkpoint
• upstream compressed-tensors NVFP4 support does not necessarily imply that every mixed multi-group checkpoint is converted identically
• the GGUF is not bit-identical to the source checkpoint
• FP8 tensors are converted to Q8_0, not preserved as native FP8
• embedded MTP tensors remain in source floating-point precision
• output quality and performance may differ from vLLM, SGLang, TensorRT-LLM, or other runtimes
• successful loading does not guarantee identical numerical behavior
• native sm_120a code does not prove that every NVFP4 operation uses an optimal FP4 Tensor Core kernel
• batch-1 token generation may not benefit as much as prompt processing or larger matrix workloads
• future converter, kernel, and dispatch changes may alter compatibility and performance

Compatibility

A recent llama.cpp build with support for the following is required:

• Qwen3.6
• GGUF NVFP4 tensors
• native MTP speculative decoding
• external MTP draft models

Tested with:

• Windows 11
• NVIDIA GeForce RTX 5070 Ti 16 GB
• NVIDIA GeForce RTX 5060 Ti 16 GB
• llama.cpp b9967-4f37f5197
• locally compiled static CUDA build
• CUDA architecture: native sm_120a
• CUDA 13.0 toolchain
• dual-GPU layer split
• --tensor-split 14,12
• external Q4_K_M MTP-only drafter on CUDA1
• --spec-draft-n-max 4
• --spec-draft-p-min 0.60

The static CUDA build was inspected with cuobjdump and contained native sm_120a cubins. This confirms that the benchmark did not rely only on older architecture cubins or PTX fallback.

Native sm_120a code targets the correct GPU architecture, but it does not prove that every NVFP4 operation uses an optimal FP4 Tensor Core kernel. Runtime performance still depends on tensor type, operator shape, kernel dispatch, device placement, and llama.cpp implementation details.

Recommended Usage

The embedded MTP tensors are usable, but on the tested dual-GPU system an external Q4_K_M MTP-only drafter was substantially faster while preserving almost the same aggregate acceptance rate.

Recommended: external Q4_K_M MTP drafter

llama-server.exe ^
  -m "unsloth-Qwen3.6-27B-NVFP4.gguf" ^
  -md "Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf" ^
  --device-draft CUDA1 ^
  --host 127.0.0.1 ^
  --port 18010 ^
  -np 1 ^
  --threads 12 ^
  --threads-batch 16 ^
  --split-mode layer ^
  --tensor-split 14,12 ^
  --ctx-size 81920 ^
  --no-mmap ^
  -ngl -1 ^
  --flash-attn on ^
  --jinja ^
  --ubatch-size 512 ^
  --batch-size 2048 ^
  --no-warmup ^
  --temp 0.6 ^
  --top-k 20 ^
  --top-p 0.80 ^
  --min-p 0.0 ^
  --presence-penalty 1.5 ^
  --repeat-penalty 1.0 ^
  --fit off ^
  --reasoning off ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 4 ^
  --spec-draft-p-min 0.60

Embedded native MTP

llama-server.exe ^
  -m "unsloth-Qwen3.6-27B-NVFP4.gguf" ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 4 ^
  --spec-draft-p-min 0.60 ^
  --flash-attn on ^
  -ngl -1

The embedded path may consume more bandwidth because the MTP tensors remain at higher precision. Performance also depends on how llama.cpp places the embedded MTP tensors across multiple GPUs.

Conversion

At the time this GGUF was created, the upstream llama.cpp converter stopped on this checkpoint with:

NotImplementedError: Can't handle multiple config groups for compressed-tensors yet

Upstream llama.cpp later added compressed-tensors NVFP4 conversion support in PR #21095. That support normalizes compressed-tensors NVFP4 metadata and scales into the existing ModelOpt-compatible conversion path.

This checkpoint additionally contains a mixed FP8 + NVFP4 configuration and unquantized MTP tensors. The historical local extension used for the uploaded file specifically handled that mixed layout:

• compatible nvfp4-pack-quantized tensors were exported as native GGUF NVFP4
• supported FP8 tensors were dequantized and stored as Q8_0 with --fp8-as-q8
• tensors without compressed scales, including embedded MTP tensors, retained their source floating-point precision

Users converting the model again should use a recent upstream llama.cpp revision and verify the resulting tensor layout. Exact output may differ from the historical local conversion used to produce this repository.

Example conversion command:

python convert_hf_to_gguf.py \
  /path/to/unsloth-Qwen3.6-27B-NVFP4 \
  --outfile unsloth-Qwen3.6-27B-NVFP4.gguf \
  --outtype auto \
  --verbose

Expected mixed layout:

NVFP4-packed compatible tensors -> native GGUF NVFP4
FP8 tensors                      -> dequantized and stored as Q8_0
embedded MTP tensors             -> BF16 / source floating-point precision

No neural-network weights were retrained.

Verifying the GGUF

Inspect the result with a recent llama-gguf build:

.\llama-gguf.exe .\unsloth-Qwen3.6-27B-NVFP4.gguf r

Expected tensor examples:

tensor: name = blk.0.ffn_down.weight,       type = nvfp4
tensor: name = blk.0.ffn_down.scale,        type = f32
tensor: name = blk.0.ffn_down.input_scale,  type = f32

The file should also contain Q8_0 tensors corresponding to source FP8 weights and floating-point tensors corresponding to the embedded MTP layers.

MTP Server Benchmark

The earlier server-based mtp-bench workload primarily compared embedded and external MTP execution on this experimental GGUF.

Aggregate Results

Aggregate throughput is total predicted tokens divided by total wall time.

The NVIDIA row is only a local comparison. Different conversion layouts, quantization choices, embedded MTP weights, and runtime behavior prevent it from being a controlled model-quality comparison.

External Q4_K_M MTP Results

code_python        pred= 192 draft= 161 acc= 142 rate=0.882 tok/s=56.7
code_cpp           pred=  53 draft=  47 acc=  41 rate=0.872 tok/s=59.8
explain_concept    pred= 192 draft= 134 acc= 103 rate=0.769 tok/s=34.0
summarize          pred=  48 draft=  42 acc=  33 rate=0.786 tok/s=47.3
qa_factual         pred= 192 draft= 154 acc= 127 rate=0.825 tok/s=44.4
translation        pred=  17 draft=  17 acc=  13 rate=0.765 tok/s=40.3
creative_short     pred=  37 draft=  19 acc=  18 rate=0.947 tok/s=29.0
stepwise_math      pred= 192 draft= 158 acc= 140 rate=0.886 tok/s=54.5
json_output        pred= 192 draft= 155 acc= 149 rate=0.961 tok/s=67.8
long_reasoning     pred= 192 draft= 155 acc= 132 rate=0.852 tok/s=47.9
repeat_pattern     pred= 192 draft= 152 acc= 152 rate=1.000 tok/s=73.8
code_completion    pred= 192 draft= 166 acc= 145 rate=0.874 tok/s=62.4
long_code_review   pred= 192 draft= 141 acc= 109 rate=0.773 tok/s=35.7

Aggregate:
  requests:            13
  predicted tokens:    1883
  draft tokens:        1501
  accepted tokens:     1304
  acceptance rate:     0.8688
  total wall time:     42.12 s

Embedded MTP Results

code_python        pred= 192 draft= 159 acc= 140 rate=0.880 tok/s=37.9
code_cpp           pred=  53 draft=  47 acc=  41 rate=0.872 tok/s=39.8
explain_concept    pred= 192 draft= 126 acc= 100 rate=0.794 tok/s=26.5
summarize          pred=  48 draft=  41 acc=  33 rate=0.805 tok/s=34.0
qa_factual         pred= 192 draft= 157 acc= 130 rate=0.828 tok/s=32.7
translation        pred=  17 draft=  18 acc=  13 rate=0.722 tok/s=28.1
creative_short     pred=  37 draft=  20 acc=  19 rate=0.950 tok/s=25.0
stepwise_math      pred= 192 draft= 157 acc= 140 rate=0.892 tok/s=37.9
json_output        pred= 192 draft= 155 acc= 149 rate=0.961 tok/s=45.5
long_reasoning     pred= 192 draft= 158 acc= 132 rate=0.835 tok/s=34.4
repeat_pattern     pred= 192 draft= 155 acc= 152 rate=0.981 tok/s=48.5
code_completion    pred= 192 draft= 159 acc= 149 rate=0.937 tok/s=44.0
long_code_review   pred= 192 draft= 145 acc= 110 rate=0.759 tok/s=28.0

Aggregate:
  requests:            13
  predicted tokens:    1883
  draft tokens:        1497
  accepted tokens:     1308
  acceptance rate:     0.8737
  total wall time:     57.46 s

Native sm_120a Mixed-Task Comparison

A second benchmark used a locally compiled native sm_120a build:

build: b9967-4f37f5197
main GPU: NVIDIA GeForce RTX 5070 Ti 16 GB
secondary GPU: NVIDIA GeForce RTX 5060 Ti 16 GB
split mode: layer
tensor split: 14,12
external draft model: Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf
draft device: CUDA1
speculative decoding: draft-mtp
draft max: 4
draft probability minimum: 0.60
temperature: 0
context size: 2048
maximum generated tokens: 1024

The workload covered translation, tagging, styled generation, Objective-C, LiveKit, tool-use prompting, KL divergence, Swift concurrency, release-note translation, and knowledge-graph planning.

Average Performance

Relative results:

• NVIDIA NVFP4 was approximately 3.4% faster than the experimental Unsloth conversion in prompt processing
• NVIDIA NVFP4 was approximately 3.0% faster in generation
• Q4_K_M was approximately 5.9% faster than NVIDIA NVFP4 in prompt processing
• Q4_K_M was approximately 5.4% faster than NVIDIA NVFP4 in generation
• Q4_K_M was approximately 9.9% faster than the experimental conversion in prompt processing
• Q4_K_M was approximately 8.6% faster than the experimental conversion in generation

These results apply only to this dual-GPU configuration and should not be generalized to other GPUs, single-GPU execution, larger batches, or different context lengths.

Per-Prompt Generation Throughput

Values are generation tokens per second.

Q4_K_M was faster in most tasks, while the NVFP4 models occasionally matched or exceeded it on individual prompts. Short single-prompt results are sensitive to output length, token distribution, speculative-decoding acceptance, and runtime scheduling.

Performance Interpretation

The native sm_120a results show that missing native Blackwell cubins were not the only cause of the earlier NVFP4 performance.

Even with verified native sm_120a code:

• the experimental mixed NVFP4 model did not outperform Q4_K_M
• the NVIDIA NVFP4 comparison also remained slightly slower than Q4_K_M
• the NVIDIA version was only modestly faster than this community conversion

Likely contributing factors include:

• only part of this GGUF uses native NVFP4 tensors
• source FP8 tensors are represented as Q8_0
• some tensors remain BF16 or another floating-point type
• mixed tensor formats require different kernels
• native Blackwell NVFP4 acceleration may benefit prompt processing and larger matrix workloads more than batch-1 token generation
• token-by-token decode may be limited by memory bandwidth, kernel-launch overhead, and small matrix shapes rather than peak Tensor Core FLOPS
• dual-GPU layer splitting introduces synchronization and peer-transfer costs
• the external draft model shares GPU resources with part of the main model
• Q4_K_M CUDA kernels and layouts are more mature in llama.cpp
• NVFP4 kernel dispatch and operator coverage are still evolving

Native sm_120a support improves compatibility and targets the correct architecture, but it does not guarantee an end-to-end NVFP4 speedup.

Quality Notes

The mixed-task outputs did not show a clear overall quality advantage for any NVFP4 GGUF.

The NVIDIA and experimental Unsloth NVFP4 outputs were generally in the same quality range:

• some NVIDIA answers were technically better
• some introduced unsupported factual details
• both produced plausible-looking but incorrect framework-specific code in some tests
• translation and short-form writing quality were broadly similar

The Q4_K_M comparison model also produced both strong answers and confident technical or factual errors.

This is not a controlled quantization-quality comparison because the tested files do not all use identical model weights:

• the Q4_K_M comparison uses an uncensored-heretic-v2 variant
• the experimental NVFP4 GGUF originates from the Unsloth checkpoint
• the NVIDIA NVFP4 GGUF may use a different conversion and quantization path

A valid quality comparison would require:

• the same original checkpoint
• identical tokenizer and chat template
• identical prompts and sampling parameters
• deterministic seeds
• reference logits from the unquantized model
• KL divergence, cross-entropy, perplexity, or token-agreement measurements
• repeated runs for non-deterministic decoding

Treat this benchmark primarily as a local runtime and practical-output comparison, not proof that one quantization format has better model quality.

Benchmark Observations

Embedded versus external MTP

On the tested system:

• the external Q4_K_M drafter reduced wall time from 57.46 to 42.12 seconds
• wall time was approximately 26.7% lower
• aggregate throughput rose from about 32.77 to 44.71 tok/s
• acceptance changed only slightly, from 87.37% to 86.88%
• predictable tasks such as JSON, repeated patterns, code completion, and stepwise reasoning benefited the most

The results suggest that an external low-precision MTP-only draft model can be faster than embedded higher-precision MTP tensors on this dual-GPU setup.

They do not prove that precision alone is responsible. Device placement, synchronization, graph construction, memory bandwidth, and backend behavior may also contribute.

NVFP4 versus Q4_K_M

In the native sm_120a mixed-task test:

• experimental mixed NVFP4 averaged 37.71 generation tok/s
• NVIDIA NVFP4 averaged 38.85 generation tok/s
• Q4_K_M averaged 40.94 generation tok/s

The NVIDIA NVFP4 GGUF was modestly faster than this conversion, indicating that the experimental conversion is reasonably close to the comparison implementation.

Neither NVFP4 model provided a clear end-to-end generation advantage over Q4_K_M on this setup. At the tested llama.cpp revision, Q4_K_M remains the more practical choice when maximum interactive generation speed is the priority.

Reproducibility Notes

Performance is highly dependent on:

• llama.cpp commit and CUDA build
• GPU model and PCIe topology
• tensor split
• context size
• KV-cache type
• flash-attention support
• sampling parameters
• draft length and probability threshold
• embedded versus external draft model
• draft-device placement

The MTP benchmark does not include a non-speculative baseline and should not be interpreted as a universal MTP speedup ratio.

The ten-prompt CLI comparison was run once per prompt. It is useful for practical A/B testing but is not statistically rigorous.

Short responses are particularly sensitive to:

• CUDA graph warm-up
• model loading and initialization
• GPU clock state
• output length
• end-of-sequence timing
• speculative-decoding acceptance
• Windows background activity

For regression testing, run each prompt at least three times and compare median generation throughput. Prompt throughput, generation throughput, and end-to-end output throughput should be treated as separate metrics.

Conclusion

This conversion demonstrates that the mixed FP8 + NVFP4 compressed-tensors checkpoint can be exported into a loadable GGUF while preserving compatible tensors as native NVFP4.

Upstream llama.cpp now supports compressed-tensors NVFP4 conversion through PR #21095, while PR #22196 provides native Blackwell CUDA execution for applicable NVFP4 matrix operations. Those additions do not make this uploaded mixed-layout GGUF fully NVFP4, nor do they guarantee that batch-1 generation uses the fastest possible FP4 path for every operation.

The model is functional and performs within a few percent of the tested NVIDIA NVFP4 GGUF comparison.

Current native sm_120a llama.cpp testing does not show a practical generation-speed advantage over Q4_K_M. On the tested dual-GPU system, Q4_K_M remained approximately 5-9% faster in average generation throughput.

The most effective optimization observed was not the main weight format, but replacing the embedded higher-precision MTP tensors with an external Q4_K_M MTP-only draft model.

Future llama.cpp improvements to small-batch and token-generation NVFP4 kernels, operator coverage, dispatch behavior, converter support, and multi-GPU execution may change these results.

Credits

• Qwen Team / Alibaba Cloud — Qwen3.6-27B
• Unsloth — mixed-precision NVFP4 source checkpoint
• ggml-org — llama.cpp, GGUF, compressed-tensors conversion, NVFP4 inference, and MTP support
• NVIDIA — NVFP4 format and related quantization work

License

The source model is distributed under the Apache License 2.0.

Review the upstream unsloth/Qwen3.6-27B-NVFP4 and Qwen/Qwen3.6-27B model cards before redistribution or commercial use.

This repository is an unofficial, experimental community conversion and is not endorsed by Unsloth, Qwen, NVIDIA, or the llama.cpp project.