CodeThink-V4-Qwen3-4B-Mix GGUF

This repository provides GGUF quantizations for modrill/CodeThink-V4-Qwen3-4B-Mix.

CodeThink-V4-Qwen3-4B-Mix is a code reasoning model built on Qwen/Qwen3-4B-Base fine-tuned using a Mix Distillation recipe (0.2 traces from Qwen3-30B-A3B-Thinking and 0.8 traces from Qwen3-4B-Thinking across 4,155 competitive programming and code reasoning problems). The final LoRA adapter was merged into full weights at the 2-epoch endpoint.

All GGUF files were converted from the original safetensors weights using llama.cpp at native BF16 precision, followed by quantization into standard k-quant variants.

---

Available Files and Quantizations

---

Technical Specifications

• Architecture: Qwen3 (Qwen3ForCausalLM)
• Parameter Count: 4.41B
• Context Window: 32,768 tokens
• Attention Configuration: 32 attention heads, 8 key-value heads, head dimension 128
• Base Precision: bfloat16
• License: Apache-2.0

---

Prompt Format

This model uses the standard ChatML / Qwen conversation format, supporting explicit <think> reasoning blocks:

<|im_start|>system
You are a helpful coding assistant.<|im_end|>
<|im_start|>user
Write a Python function to check if a number is prime.<|im_end|>
<|im_start|>assistant
<think>
Need to verify if n <= 1, then check divisors up to sqrt(n).
</think>

def is_prime(n: int) -> bool:
    if n <= 1:
        return False
    if n <= 3:
        return True
    if n % 2 == 0 or n % 3 == 0:
        return False
    i = 5
    while i * i <= n:
        if n % i == 0 or n % (i + 2) == 0:
            return False
        i += 6
    return True
<|im_end|>

---

Quickstart Guide

Option 1: llama.cpp CLI

Run local interactive inference directly with llama-cli:

# Run 4-bit quantized model
llama-cli -m CodeThink-V4-Qwen3-4B-Mix-Q4_K_M.gguf \
  -p "<|im_start|>user\nWrite a quicksort implementation in Python.<|im_end|>\n<|im_start|>assistant\n" \
  -n 512 \
  -ngl 99

Option 2: llama.cpp Server

Launch an OpenAI-compatible HTTP server:

llama-server -m CodeThink-V4-Qwen3-4B-Mix-Q4_K_M.gguf \
  --port 8080 \
  -c 32768 \
  -ngl 99

Query the endpoint:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "CodeThink-V4-Qwen3-4B-Mix",
    "messages": [
      {"role": "user", "content": "Explain how Dijkstra algorithm works."}
    ],
    "temperature": 0.6
  }'

Option 3: Ollama

Create an Ollama model using a custom Modelfile:

FROM ./CodeThink-V4-Qwen3-4B-Mix-Q4_K_M.gguf

TEMPLATE """{{- if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{- end }}
{{- range .Messages }}
<|im_start|>{{ .Role }}
{{ .Content }}<|im_end|>
{{- end }}
<|im_start|>assistant
"""

PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"
PARAMETER temperature 0.6
PARAMETER top_p 0.95

Build and run with Ollama:

ollama create codethink-4b -f Modelfile
ollama run codethink-4b "Write a binary search function in Rust."

Option 4: LM Studio

1. Copy the desired .gguf file into your local LM Studio models folder.
2. Select CodeThink-V4-Qwen3-4B-Mix-Q4_K_M.gguf from the model picker.
3. Use the ChatML prompt preset with system and user markers.

---

Original Model and Citation

• Original Hugging Face Checkpoint: modrill/CodeThink-V4-Qwen3-4B-Mix
• Base Architecture Reference: Qwen/Qwen3-4B-Base
• Mix Distillation Methodology: Li et al., 2025, "Small Models Struggle to Learn from Strong Reasoners", arXiv:2502.12143