lododo-Distill-v1.0 · LoRA adapter (GGUF)
This is the LoRA adapter form (40 MB) of lododo-Distill-v1.0.
The model itself: a distilled fine-tune of the Qwen3.6-35B-A3B base (MoE, 3B active), trained on data generated by Qwen3.8-27B as the teacher — bringing the base to (and beyond) the teacher's capability while keeping the MoE speed advantage.
| Metric | Teacher: Qwen3.8-27B | Base: Qwen3.6-35B-A3B | This LoRA (f16) | Merged build |
|---|---|---|---|---|
| LCB code (AtCoder 112) | 52.7% | 50.9% | 67.9% | 69.6% |
| BFCL tool calling (100) | 53.0% | 62.0% | 66.0% | 67.0% |
| GSM8K math (500) | 95.2% | 94.4% | 96.0% | 96.0% |
| IFEval instruction (150) | 36.7% | 24.0% | 33.3% | 33.3% |
Which repo should you use?
| Your situation | Pick |
|---|---|
You already have Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf (or plan to reuse it) |
✅ this LoRA (40 MB) — only 40 MB on top |
| Starting from scratch and want the best result | merged build: lododoai/Qwen3.6-35B-A3B-Distill-Qwen3.8-27B-GGUF (69.6%, 22.85 GB) |
Mounting the LoRA is ~1.7pp below the merged build (the delta is applied at runtime instead of being baked into the weights), in exchange for 22 GB of disk/traffic.
🧰 Running this model on a small GPU? Use moefast
moefast is a small launcher that configures llama.cpp for your machine. It reads the model's GGUF
metadata, measures your free VRAM, computes the expert-cache size, context and batch parameters,
writes them into an editable config file next to the model, and starts llama-server.
The expert cache itself — keeping expert weights in host RAM and caching the hot ones in VRAM — is
llama.cpp's own (--moe-cache-mib, upstream PRs #29887/#30112). moefast makes sure it is
enabled, correctly sized, and remembered per model — the cache defaults to off in llama.cpp,
which roughly halves throughput.
git clone https://github.com/conlododo/moefast && cd moefast
./install.sh
./bin/moefast run /path/to/this-model.gguf
No patch, no fork — stock llama.cpp only. Repo: https://github.com/conlododo/moefast
Files
| File | Size | Notes |
|---|---|---|
lododo-Distill-v1.0-lora-f16.gguf |
40 MB | Recommended (f16, essentially lossless) |
Usage
# 1) download the base (must be the MTP-enabled build)
hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --local-dir .
# 2) start with the LoRA mounted
llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
--lora lododo-Distill-v1.0-lora-f16.gguf \
-ngl 99 --jinja -c 32768 --port 8080
Sampling parameters (only relevant for MTP / Strata)
The model has no special sampling requirement — set temperature / top_p as your task needs.
Only when you enable MTP speculative decoding in an engine that injects model-card sampling defaults (e.g. Strata) must the request carry:
{"temperature": 0, "presence_penalty": 0, "frequency_penalty": 0, "repeat_penalty": 1.0}
Why: Strata injects temperature=1.0, top_p=0.95, top_k=20, presence_penalty=1.5 for qwen35moe,
and its MTP gate requires temperature==0 && repeat==1 && freq==0 && present==0 — the defaults shut the
drafter off entirely (drafts accepted 0 of 0 in our tests).
llama.cpp's own --spec-type draft-mtp is not affected.
Training summary
| Item | Value |
|---|---|
| Teacher → base | Distillation data from Qwen3.8-27B → Qwen3.6-35B-A3B |
| Data | 48.73M tokens (~52k samples) |
| Method | QLoRA r=16 / α=32 (attention layers + shared experts) |
| Training | 0.5 epoch (747 steps), single uninterrupted run |
| Stability | Equal-weight average of two independent runs (reduces random-init variance) |
Known limitations
| Item | Notes |
|---|---|
| LeetCode format | 7.9% (training data is AtCoder stdin/stdout style) |
| IFEval | 33.3%, 3.4pp below the 27B teacher |
| Long context | Supports up to the base's native 256K context; usable length depends on VRAM |
License & acknowledgements
- Released under Apache-2.0, inherited from Qwen/Qwen3.6-35B-A3B.
- The base GGUF quantization comes from unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Apache-2.0).
- Not affiliated with or endorsed by the Qwen team; "Qwen" is used only to describe the base model.
- Use at your own risk; no warranty of any kind.