A community mixed-precision conversion for local inference with FreeToken, tested on an RTX 5070 Ti 16 GB and 64 GB system RAM.
This release converts Empero's Qwen3.8-35B-A3B-Distill to NVFP4/FP8 while retaining selected components in BF16. It includes the converted weights, a portable launcher, exact runtime settings, reproduction instructions, checksums, and evaluation records.
The goal was practical: use a 35B MoE model on the workstation I already own, keep a large reusable prefix cache, and load or unload the model through FreeToken Desktop. Expert weights live in system RAM with a GPU expert cache. The approximately 24.19 GB checkpoint does not fit entirely in 16 GB VRAM.
| At a glance | Release details |
|---|---|
| Model | Approximately 35B total parameters, 3B active per token |
| Quantization | NVFP4 experts, selected FP8 attention projections, retained BF16 components |
| Tested workstation | RTX 5070 Ti 16 GB, Ryzen 9 9900X, 64 GB RAM, Windows |
| Tested runtime | FreeToken Desktop v0.2.0-beta.23; engine 0.1.3+gc8ed699cb |
| Deployment | Expert weights offloaded to RAM; raw safetensors loading |
| Native context limit | 262,144 tokens per sequence |
| Allocated attention cache | 347,136 shared token slots, displayed as 339K |
| Long prompt tested | 138,187 input tokens |
| License | Apache-2.0, following the source declaration |
This is an independent community conversion. It is not an official Alibaba or NVIDIA release. The recorded tests establish that this checkpoint ran with the documented setup; they do not establish a quality advantage over the original Qwen3.6 model or the BF16 source.
The steps below were tested with FreeToken Desktop v0.2.0-beta.23 on Windows. Other runtimes and GPUs have not been validated by this release.
Download this repository to a permanent local folder. For example, with the Hugging Face CLI:
hf download qualitymaterial/Qwen3.8-35B-A3B-Distill-NVFP4-FreeToken --local-dir 'C:\Models\Qwen3.8-35B-A3B-Distill-NVFP4-FreeToken'
In FreeToken Desktop, open Models and select the local model-folder import control.
Select the downloaded folder containing config.json and the safetensors files, then start the imported model.
In Console, enter the settings below and select Apply.
| Setting | Tested value |
|---|---|
| KV cache | 347,136 tokens |
| Expert cache | 512 slots |
| Mamba state slots | 24 |
FreeToken displays the attention cache as 339K and saves the applied settings for the model's next launch. Use Models → Start/Stop to load or unload it.
Reduce the cache settings if your available memory differs. Desktop applications also consume memory, so the same nominal hardware specifications do not guarantee the same available allocation.
339K is a shared cache pool, not a validated 339K conversation window. The native per-sequence limit is 262,144 tokens. The longest prompt tested here was 138,187 tokens; context extension beyond the native limit and a full 262K prompt were not tested.
The default local API is http://127.0.0.1:1919/v1. Query /v1/models for the runtime's served model ID. No FTW preconversion was required for the recorded tests.
Start-FreeToken-Distill.ps1 requires an installed, running FreeToken daemon. Run it from the downloaded repository folder; it defaults to that folder or accepts an explicit model path:
.\Start-FreeToken-Distill.ps1 -ModelPath 'C:\Models\Qwen3.8-35B-A3B-Distill-NVFP4-FreeToken'
.\Start-FreeToken-Distill.ps1 -Action Stop
Start can switch away from the daemon's current model. Stop stops its current model. Desktop import is still required for a library entry, and Console Apply saves settings for subsequent desktop launches.
The packaged launcher is adapted from the tested local launcher. Its parameterized wrapper was syntax-checked, but was not separately exercised in a new model restart.
freetoken-settings.json records the exact runtime settings:
{"num_pages": 347136, "moe_cache_size": 512, "num_mamba_slots": 24}
This is the request body for POST /v1/cache/rebuild on the inference API. Keep it separate from config.json, which defines the model architecture and quantization.
These are the initial recorded tests on the workstation above, not a new benchmark run. Measurements and derived statistics are in evaluation/release-stats.json.
| Test | Observed result |
|---|---|
| Short response | Passed |
| Automatic function call with JSON arguments | Passed smoke check |
| Tool-result round trip | Passed smoke check |
| Reasoning-enabled arithmetic | Correct on the tested example |
| Short prefix reuse | 3,648 cached tokens reported |
| Synthetic long prompt | 138,187 input tokens; beginning, middle, and end markers retrieved correctly |
| First long request | 107.0 seconds end to end |
| Identical-prefix repeat | 2.57 seconds end to end; 138,176 cached tokens reported |
| Short coding decode | Approximately 53–62 tokens/s in steady server log intervals |
| Desktop controls | Stop unloaded; Start reloaded successfully |
The repeated long request reused 99.992% of the input tokens and completed 41.63× faster end to end in this synthetic test. That ratio primarily reflects avoiding repeated prefix processing. It is not a general decode-speed improvement.
The coding speed is a log-based observation for a short response, not an aggregate benchmark. That response also included Markdown fences despite an instruction requesting bare code. Successful basic tool calls do not establish reliable multi-step agent performance.
| Metric | Value | Evidence |
|---|---|---|
| Attention cache pool | 347,136 token slots, displayed as 339K | Engine cache geometry |
| Attention cache bytes per token | 20,480 bytes = 20 KiB | Engine cache geometry |
| Attention cache allocation | 6.621 GiB | Calculated from geometry |
| GPU expert cache | 512 slots across 40 MoE layers | Engine cache geometry |
| Expert cache allocation | 0.847 GiB | Calculated from geometry |
| Recurrent state slots | 24 | Engine cache geometry |
| Recurrent state allocation | 1.439 GiB | Calculated from geometry |
| Combined configured caches | 8.907 GiB | Sum of the three allocations; excludes weights and runtime buffers |
| Long-prompt cached fraction on repeat | 99.992% | 138,176 / 138,187 input tokens |
| End-to-end repeat speedup | 41.63× | 107.0166 / 2.57092 seconds |
No maximum stable concurrency test was performed. The 24 recurrent-state slots do not imply that 24 simultaneous long conversations fit.
Scripts and synthetic-response records are in evaluation/. synthetic-long-prompt.json contains the exact request body reconstructed deterministically from the original test script: 8,192 reference rows, marker substitutions, messages, and generation settings.
This request file was reconstructed after the test rather than captured as a request log. The server reported 138,187 prompt tokens for that request; the JSON file's size is not its token count. synthetic-long-prompt.sha256 verifies the reconstructed request file.
No broad benchmark or matched real-world DeepSeek Harness comparison has been completed. Quantization accuracy relative to the BF16 source has not been measured.
I wanted a local model I could use for sustained agent work in DeepSeek Harness Desktop on the hardware I already own. In my local testing, FreeToken gave me the most useful combination of performance and cache capacity. That is my experience with this setup, not a controlled comparison against every inference engine.
My working Qwen3.6 setup already had a 339K cache pool. I wanted to try the newer community distillation without giving up that allocation or buying a larger GPU. I also wanted a permanent checkpoint I could load, unload, and share with enough detail for someone else to reproduce the setup.
With MoE experts in RAM, the whole checkpoint does not have to fit in VRAM. GPU memory still has to cover resident tensors, working buffers, the expert cache, recurrent state, and the attention cache. Increasing one cache leaves less room for the others. This conversion let me keep a relatively small GPU expert cache and reserve more VRAM for reusable context. The largest possible cache is not necessarily the fastest configuration.
RunPod supplied temporary conversion hardware. Daily inference and all tests reported here ran locally; continued GPU rental is not required to use this checkpoint with the documented local setup.
The architecture is inherited from Qwen3.6-35B-A3B: approximately 35B total parameters, 3B active per token, and hybrid linear/full attention. Empero performed the distillation. This release changes weight precision, not training or architecture.
| Component | Stored precision |
|---|---|
| Routed and shared experts | NVFP4, weight-only W4A16 |
| Selected full/linear attention projections | FP8, with dynamic activation quantization configuration |
| Embeddings, output head, routers, norms, sensitive recurrent projections, convolution, vision, and MTP | Retained BF16 per recipe |
The conversion uses data-free post-training quantization. It does not use calibration-based quantization or quantization-aware training.
| Conversion detail | Value |
|---|---|
| Source | empero-ai/Qwen3.8-35B-A3B-Distill |
| Source revision | bcc2dbe2f21b213625df2dc1a5a690212373af07 |
| Recipe | Qwen DGX Spark conversion project |
| Conversion dependency | llmcompressor==0.14.0 |
| Conversion GPU | RTX PRO 4500 Blackwell, via RunPod |
| Output | 21 main safetensors shards plus an MTP shard |
| Checkpoint payload | 24,193,559,244 bytes, approximately 24.19 GB / 22.53 GiB, including tokenizer/configuration files |
See REPRODUCE.md for the pinned recipe reference and conversion steps. The full transitive dependency environment was not captured, so byte-identical reproduction is not guaranteed. SHA256SUMS verifies the original 32 checkpoint payload files; release documentation is outside that manifest.
The recipe repository did not declare a license when checked on 2026-10-02. Its script is linked rather than redistributed here. Contact the recipe author or the DGX-Spark project if you plan to redistribute the recipe itself.
| Specification | Verified value |
|---|---|
| GPU | NVIDIA GeForce RTX 5070 Ti |
| GPU memory reported by driver | 16,303 MiB, approximately 15.92 GiB |
| CPU | AMD Ryzen 9 9900X, 12 cores / 24 logical processors |
| System RAM | 64 GB installed; FreeToken reports approximately 61.6 GiB usable |
| Operating system | Windows |
The payload alone exceeds GPU memory. This deployment uses system RAM and the GPU together. Available memory, background applications, and cache settings affect whether another machine can reproduce the same allocation.
Weights follow the source's Apache-2.0 declaration. See LICENSE and NOTICE.md.
model-00001-of-00021.safetensors 2.6 GBmodel-mtp.safetensors 1.7 GBmodel-00021-of-00021.safetensors 1.2 GBmodel-00002-of-00021.safetensors 1.1 GBmodel-00007-of-00021.safetensors 1.0 GBmodel-00009-of-00021.safetensors 1.0 GBmodel-00011-of-00021.safetensors 1.0 GBmodel-00013-of-00021.safetensors 1.0 GBmodel-00015-of-00021.safetensors 1.0 GBmodel-00017-of-00021.safetensors 1.0 GBmodel-00019-of-00021.safetensors 1.0 GBmodel-00003-of-00021.safetensors 1.0 GBmodel-00005-of-00021.safetensors 1.0 GBmodel-00008-of-00021.safetensors 946 MBmodel-00010-of-00021.safetensors 946 MBmodel-00012-of-00021.safetensors 946 MBmodel-00014-of-00021.safetensors 946 MBmodel-00016-of-00021.safetensors 946 MBmodel-00018-of-00021.safetensors 946 MBmodel-00020-of-00021.safetensors 946 MBmodel-00006-of-00021.safetensors 946 MBmodel-00004-of-00021.safetensors 946 MBempero-ai/Qwen3.8-35B-A3B-Distill