中文Sign up

NasledieLab/Qwen3.5-0.8B-GPT6-Luna-Distill-Karpy

Source: 🤗image-text-to-textapache-2.0800M activated4.2 GBGGUF✓ 2 checksumsupdated 23d ago
or submit a torrent

Qwen3.5-0.8B-GPT6-Luna-Distill-Karpy (GGUF, fixed layout)

Distill of unsloth/Qwen3.5-0.8B (Qwen3_5ForConditionalGeneration, text 1024/24L + vision 768/12L, ctx 262144). This repo layout fixes the broken single-file GGUF so one command works everywhere once the GGUF binary is rebuilt.

Status: FIXED. Rebuilt with official llama.cpp converter (arch qwen35, 320 text tensors + 154 mmproj tensors, Q4_K_M 505 MB + mmproj-F16 196 MB). validate_gguf.py prints PASS on both files. Old files below are deleted.

Expected files after rebuild

Qwen3.5-0.8B-GPT6-Luna-Distill-Karpy-Q4_K_M.gguf  # text, arch qwen35, quantized, ~0.5 GB
mmproj-F16.gguf                                   # vision encoder, F16
README.md
Modelfile                                         # local `ollama create` only
template                                          # Ollama HF custom template (Go)
system                                            # Ollama HF default system prompt
params                                            # Ollama HF sampling params (JSON)
.gitattributes
config.json                                       # source reference only
validate_gguf.py
REBUILD_GGUF.ps1 / rebuild_gguf.sh

Delete on Hub: model-q4_k_m.gguf, gguf-metadata.json, tokenizer.json (embedded in GGUF).

llama.cpp (single command, text)

llama cli -hf NasledieLab/Qwen3.5-0.8B-GPT6-Luna-Distill-Karpy:Q4_K_M -p "Hello"
llama serve -hf NasledieLab/Qwen3.5-0.8B-GPT6-Luna-Distill-Karpy:Q4_K_M

Vision (after rebuild, single command line):

llama serve -hf NasledieLab/Qwen3.5-0.8B-GPT6-Luna-Distill-Karpy:Q4_K_M --mmproj mmproj-F16.gguf

If -hf does not auto-fetch mmproj, download both files first:

hf download NasledieLab/Qwen3.5-0.8B-GPT6-Luna-Distill-Karpy Qwen3.5-0.8B-GPT6-Luna-Distill-Karpy-Q4_K_M.gguf --local-dir .
hf download NasledieLab/Qwen3.5-0.8B-GPT6-Luna-Distill-Karpy mmproj-F16.gguf --local-dir .
llama-server -m Qwen3.5-0.8B-GPT6-Luna-Distill-Karpy-Q4_K_M.gguf --mmproj mmproj-F16.gguf

Reference working repos: ggml-org/Qwen3.5-0.8B-GGUF (text-only, arch qwen35, ctx 262144), unsloth/Qwen3.5-0.8B-GGUF (text + mmproj-*.gguf).

Ollama (exact command requested)

ollama run hf.co/NasledieLab/Qwen3.5-0.8B-GPT6-Luna-Distill-Karpy:Q4_K_M

Notes:

  • hf.co flow ignores Modelfile; it picks the GGUF by :Q4_K_M tag and uses embedded tokenizer.chat_template, or template / system / params files in this repo.
  • Text works after rebuild. Vision in Ollama HF requires a proper single-file qwen35 VL GGUF (no 5-D tensors). Until then use llama.cpp / LM Studio for vision.
  • Local build (same folder as GGUF):
ollama create karpy-luna -f Modelfile
ollama run karpy-luna

LM Studio / Jan / Docker Model Runner

  • LM Studio: search this repo or paste HF URL, pick Q4_K_M. For vision attach mmproj-F16.gguf when prompted.
  • Jan: jan://models/huggingface/NasledieLab/Qwen3.5-0.8B-GPT6-Luna-Distill-Karpy
  • Docker: docker model run hf.co/NasledieLab/Qwen3.5-0.8B-GPT6-Luna-Distill-Karpy:Q4_K_M

Requires recent llama.cpp / Ollama with qwen35 support. Update if load fails with unknown architecture.

What was broken

  1. general.architecture=llama with HF names (model.language_model.*, model.visual.*). Must be qwen35.
  2. 5-D model.visual.patch_embed.proj.weight. GGML max 4 dims. Proper converter splits text + mmproj and reshapes.
  3. All 473 tensors F32 (3.41 GB), mislabeled Q4_K_M. Real Q4_K_M 0.8B is ~0.5 GB.
  4. Only 8 KVs; ctx 4096 / hidden 2048 / layers 16 / vocab 32000 vs source 262144 / 1024 / 24 / 248320. Missing tokenizer + chat template.
  5. gguf-metadata.json described the broken file and falsely claimed Ollama/llama.cpp/vLLM/LM Studio compat.
  6. Modelfile ChatML template missed <|vision_start|>, <|image_pad|>, <think>, tool calls.

Rebuild (owner step, required)

.\REBUILD_GGUF.ps1 -SourceDir D:\path\to\karpy-safetensors -OutDir .
python .\validate_gguf.py Qwen3.5-0.8B-GPT6-Luna-Distill-Karpy-Q4_K_M.gguf

See PR_DESCRIPTION.md for upload/delete list.