Qwen3.8-4B-Distill-Heretic-Abliterated-MLX-4bit

MLX 4-bit quantized version of insraq/Qwen3.5-4B-EmperoAI-Qwen3.8-Distill-Heretic-Abliterated, with full vision weights preserved.

Model lineage

Qwen3.5-4B (base, Alibaba)
  └─ empero-ai/Qwen3.8-4B-Distill (distilled from Qwen3.8, vision-capable, 5B params incl. ~1B vision encoder)
      └─ insraq/Qwen3.5-4B-EmperoAI-Qwen3.8-Distill-Heretic-Abliterated (Heretic v1.4.0 abliteration, refusals 6/100 vs 99/100)
          └─ This repo (MLX 4-bit quantization, vision weights preserved via mlx_vlm)

Why this exists

Most MLX quantizations of Qwen3.8 distilled models on HuggingFace are blind — they were converted with mlx_lm.convert, which silently drops all vision encoder weights, leaving a text-only model that still reports image-text-to-text in its config but cannot actually process images.

This model was converted with mlx_vlm.convert (not mlx_lm.convert), which correctly preserves the vision tower:

Conversion details

Reproduce

# Install mlx_vlm (not mlx_lm!)
pip install mlx-vlm jinja2

# Convert with vision weights preserved
python -m mlx_vlm convert \
  --hf-path insraq/Qwen3.5-4B-EmperoAI-Qwen3.8-Distill-Heretic-Abliterated \
  --mlx-path Qwen3.8-4B-Distill-Heretic-Abliterated-MLX-4bit \
  -q --q-bits 4 --q-group-size 64 \
  --trust-remote-code

# Copy preprocessor_config.json from any Qwen3.5 vision model
# (mlx_vlm doesn't generate it; mlx-serve needs it for image processing)
cp /path/to/Qwen3.5-4B-MLX-4bit/preprocessor_config.json \
   Qwen3.8-4B-Distill-Heretic-Abliterated-MLX-4bit/

Benchmark vs Qwen3.5-4B-MLX-4bit (mlx-community)

Both models tested via mlx-serve 26.8.9 on Apple M2 Ultra (192GB), same API, same prompts, temperature=0.1.

Speed

Speeds are essentially identical — both are 4B dense models with the same architecture (Qwen3_5ForConditionalGeneration, 32 layers, 2560 hidden). The Heretic abliteration does not affect inference speed.

Vision accuracy

Text quality

Abliteration (refusal rate)

The Heretic v1.4.0 abliteration is effective: 0/3 refusals vs 1/3 for the base Qwen3.5-4B. The base model still has residual safety alignment on "how-to" instructions, while the abliterated version handles all prompts without refusal.

Usage

With mlx-serve

mlx-serve serve --model-dir ./models --port 11234

import requests
resp = requests.post("http://localhost:11234/v1/chat/completions", json={
    "model": "Qwen3.8-4B-Distill-Heretic-Abliterated-MLX-4bit",
    "messages": [{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
        {"type": "text", "text": "Describe this image."}
    ]}]
})

With mlx_vlm (Python)

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

model, processor = load("yachen4ever/Qwen3.8-4B-Distill-Heretic-Abliterated-MLX-4bit")
config = load_config("yachen4ever/Qwen3.8-4B-Distill-Heretic-Abliterated-MLX-4bit")

Acknowledgments

• empero-ai — original Qwen3.8 distilled models with vision
• insraq — Heretic v1.4.0 abliterated version
• ml-explore/mlx-vlm — MLX vision model conversion and inference
• Blaizzy/mlx-serve — OpenAI-compatible MLX server with vision support

License

Apache 2.0 (inherited from Qwen3.5/Qwen3.8 base models)