English注册

TeichAI/Qwen3.8-27B-Fable-Distill-GGUF

🤗 Hugging Face 来源image-text-to-textapache-2.0激活 27B271 GBGGUF✓ 17 个校验和今天更新
一条命令提交

在你的模型文件夹旁边运行它。它会制作种子、将文件与 Hugging Face 比对，然后提交。你只需开始做种，并粘贴你账户中的密钥。它只读取你的文件，绝不修改。如果愿意，可以先阅读脚本。

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo TeichAI/Qwen3.8-27B-Fable-Distill-GGUF ./model-folder
需要做种者 →

Qwen3.8-27B-Fable-Distill — GGUF

Model ARC Challenge ARC Challenge (Easy) BoolQ
Qwen3.8-27B 0.591 0.782 0.896
Qwen3.8-27B-Fable-Distill 0.637 0.832 0.911

As always, big thank you to @nightmedia for the benchmarks

GGUF conversions of TeichAI/Qwen3.8-27B-Fable-Distill, a BF16 finetune of Qwen3.8-27B (base: Qwen/Qwen3.8-27B) trained with Unsloth + TRL.

The model was trained on a public set of chat and agent traces from Fable 5 as well as a much larger corpus of private personal Fable 5 data.

Converted with llama.cpp b6b4344e.

MTP head kept at BF16

This model ships a multi-token-prediction (nextn) head, and every quant here keeps that head unquantized at BF16 while the other 64 layers are quantized normally:

qwen35.block_count          = 65      # 64 transformer layers + 1 MTP layer
qwen35.nextn_predict_layers = 1
blk.64.*                    = bf16    # 424.7M params, left alone

Files

File Bits Size Notes
BF16/…-BF16-*.gguf 16 ~55 GB Full precision, split into shards. Convert your own quants from this.
…-Q8_0.gguf 8 ~29 GB Near-lossless.
…-Q6_K.gguf 6 ~23 GB Very close to Q8_0 at meaningfully smaller size.
…-Q5_K_M.gguf 5 ~20 GB Strong quality/size balance.
…-Q5_K_S.gguf 5 ~19 GB
…-Q4_K_M.gguf 4 ~17 GB Recommended default for most users.
…-Q4_K_S.gguf 4 ~16 GB Slightly smaller than Q4_K_M.
…-IQ4_NL.gguf 4 ~16 GB Non-linear 4-bit.
…-IQ4_XS.gguf 4 ~16 GB Smallest of the 4-bit family.
…-Q3_K_L.gguf 3 ~15 GB
…-Q3_K_M.gguf 3 ~14 GB
…-Q3_K_S.gguf 3 ~13 GB
…-Q2_K.gguf 2 ~11 GB Noticeable quality loss.
mmproj-F32.gguf 32 1.8 GB Vision projector, full precision.
mmproj-BF16.gguf 16 0.9 GB Vision projector, bfloat16.
mmproj-F16.gguf 16 0.9 GB Vision projector, float16. Fine for nearly everyone.

Usage

Text:

llama-cli -m Qwen3.8-27B-Fable-Distill-Q4_K_M.gguf -c 8192 -p "Hello"

Vision — pass the projector alongside the model:

llama-mtmd-cli -m Qwen3.8-27B-Fable-Distill-Q4_K_M.gguf \
               --mmproj mmproj-F16.gguf \
               --image photo.jpg -p "Describe this image."

Server:

llama-server -m Qwen3.8-27B-Fable-Distill-Q4_K_M.gguf --mmproj mmproj-F16.gguf

Server + MTP:

llama-server -m Qwen3.8-27B-Fable-Distill-Q4_K_M.gguf --mmproj mmproj-F16.gguf --spec-type draft-mtp --spec-draft-n-max 3

Notes

  • The model is multimodal (image-text-to-text). Without an mmproj-*.gguf you get a text-only model. Three precisions are provided; F16 is the usual choice, BF16 matches the source weights' dtype, and F32 is there if you want the projector left entirely unquantized.
  • Qwen3.5-family chat template with thinking support: it accepts enable_thinking and a reasoning_effort of low, medium or xhigh (the template's own default is xhigh, which thinks at length every turn).
  • Base model sampling recommendations: temperature 1.0, top_p 0.95, top_k 20.

The data for this model was easily formatted, validated, and masked using Teich

This qwen3_5 model was trained 2x faster with Unsloth and Huggingface's TRL library.