Spark-X2.6-4B Quantized

GGUF and Safetensors (FP8, NVFP4) quantized variants of XHToken/Spark-X2.6-4B for local execution, agent workflows, and reasoning.

Spark-X2.6-4B by XHToken is a lightweight 4B language model optimized for on-device agents, coding, and tool use. Featuring a hybrid-attention design with up to a 1M context window and controllable thinking budgets (low / high), it matches or outperforms much larger 9B–12B models on standard agent and coding benchmarks while remaining exceptionally fast.

Benchmark

Quantized Files

Q4_K_M is recommended for the best balance of size and quality for GGUF. NVFP4 and FP8 offer high performance on supported NVIDIA GPUs.

Usage

1. llama.cpp / llama-server (GGUF)

Interactive Chat via CLI

Run multi-turn interactive conversation directly in your terminal:

./llama-cli -m Spark-X2.6-4B-Q4_K_M.gguf \
    -cnv \
    -c 8192 \
    -n 512 \
    --temp 0.6 \
    -t 8

OpenAI-Compatible API Server

Start llama-server to serve OpenAI-compatible /v1/chat/completions endpoints:

./llama-server -m Spark-X2.6-4B-Q4_K_M.gguf \
    -c 8192 \
    --port 8080

Query via curl:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Hello! What can you help me with?"
      }
    ],
    "temperature": 0.6
  }'

2. Ollama (GGUF)

Create a Modelfile:

FROM ./Spark-X2.6-4B-Q4_K_M.gguf

TEMPLATE """{{- if .System }}<｜start▁of▁sentence｜><|System|>{{ .System }}
{{ end }}{{- range .Messages }}<｜start▁of▁sentence｜><|{{ if eq .Role "user" }}User{{ else }}Bot{{ end }}|>{{ .Content }}<｜end▁of▁sentence｜>{{- end }}{{- if .Prompt }}<｜start▁of▁sentence｜><|User|>{{ .Prompt }}<｜end▁of▁sentence｜><｜start▁of▁sentence｜><|Bot|><think>
{{ end }}"""

PARAMETER stop "<｜end▁of▁sentence｜>"
PARAMETER stop "<｜start▁of▁sentence｜>"
PARAMETER temperature 0.6

Build and run:

ollama create spark-x2.6 -f Modelfile
ollama run spark-x2.6

3. NVIDIA FP8 & NVFP4 (Safetensors)

Load the quantized weights directly via safetensors for TensorCore-accelerated pipelines (such as comfy-kitchen, vLLM, or custom inference):

import safetensors.torch
import torch

# Load FP8 (torch.float8_e4m3fn with per-tensor scales)
fp8_weights = safetensors.torch.load_file("spark-x2.6-4b-fp8.safetensors")

# Load NVFP4 (packed 4-bit with block-wise scales)
nvfp4_weights = safetensors.torch.load_file("spark-x2.6-4b-NVFP4.safetensors")

Source and Build

• Source model: XHToken/Spark-X2.6-4B
• Architecture: Spark2_5ForCausalLM (spark2_5 GGUF architecture)
• Conversion & Quantization: Upstream llama.cpp & comfy-kitchen
• License: Apache 2.0
• Checksums: SHA256SUMS