Sign up

Foresee/Qwen3.8-9B-heretic-uncensored-4bit-MTPLX

🤗 Hugging Face sourcetext-generationapache-2.09B params18 GBsafetensors✓ HF checksums availableupdated today
No torrent yet →

Qwen3.8-9B Heretic Uncensored - 4-bit MLX + MTP (MTPLX)

Fast local build of rohit267/Qwen3.8-9B-heretic-uncensored for Apple Silicon, tuned for native-MTP speculative decoding with MTPLX.

Measured 21.25 -> 40.05 decode tok/s (+88%) on an M5 24GB MacBook, lossless (exact speculative verify - the emitted tokens match the target model's own decoding).

Contents

File What it is
model.safetensors (+ index) 4-bit affine MLX trunk, group size 64 (4.5 bpw, ~5.0 GB)
mtp.safetensors Native MTP head sidecar (Qwen3.5-9B lineage)
mtp-adapter-r64.npz Trained C4 LoRA MTP adapter (rank 64) that lifts draft acceptance on this fine-tune
mtplx_runtime.json MTPLX runtime contract (depth, hidden variant, sampler)

The adapter is merged at load time; it is not baked into mtp.safetensors, so the base sidecar stays reusable.

How it was built

  1. Convert the BF16 source to 4-bit MLX: mlx_lm.convert --q-bits 4 --q-group-size 64.
  2. Graft the Qwen3.5-9B MTP sidecar, then train a rank-64 C4 LoRA adapter on hidden states captured from this 4-bit trunk (coding + math + long-code calibration). Training the adapter on the final trunk is required - an adapter calibrated on a different trunk pairs wrongly.
  3. Serve with a tape-replay GDN verify kernel at draft depth 2.

Full lever study (every quantization / depth / adapter-rank / verify-core tested): see the autoresearch repo linked below.

Run it

Requires macOS 14+, Apple Silicon, and MTPLX (brew install youssofal/mtplx/mtplx).

mtplx serve \
  --model . \
  --model-id qwen-heretic \
  --mtp --depth 2 \
  --profile sustained \
  --verify-strategy capture_commit \
  --verify-core linear-gdn-from-conv-tape \
  --mtp-adapter mtp-adapter-r64.npz --merge-mtp-adapter \
  --host 127.0.0.1 --port 8080

Serves OpenAI /v1/chat/completions and Anthropic /v1/messages (Claude Code / OMP compatible).

Notes / caveats

  • Apple Silicon + MLX only. The MTP fast path is MTPLX-specific.
  • Lossy trunk, lossless decode. 4-bit quantization changes the weights; the speculative decode itself is exact-verify, so output matches the 4-bit target model.
  • This fine-tune always opens with a <think> block; budget tokens accordingly.
  • Draft acceptance on the benchmark suite is ~0.71 at depth 2; depth 3+ does not pay on this hybrid (GatedDeltaNet) 9B. ~40 tok/s is the honest lossless ceiling on this model / this hardware.

License

Apache-2.0, inherited from the Qwen3.5-9B base.