Qwen3.8-27B — oQ2

An oMLX-optimised mixed-precision conversion of Qwen/Qwen3.8-27B, packaged for MLX-VLM and oMLX on Apple silicon.

Original model · Qwen · MLX-VLM · Apache 2.0

About this conversion

This repository contains an oQ2 mixed-precision MLX conversion of Qwen3.8-27B. The global recipe is 2-bit affine with selected sensitive modules retained at substantially higher precision. The upstream model is a dense, native vision-language model with flexible thinking control and support for text, images, and video. Its tokenizer, processor configuration, chat template, and generation configuration are preserved.

Apple-silicon performance

This checkpoint was load-tested and generation-tested on the following machine:

The decode figure is the median of three greedy runs after a 256-token Metal-kernel warm-up. Each timed run ended naturally at EOS after 181 generated tokens; the runs measured 52.53, 52.61, and 52.55 tokens/s. This is a practical local reference, not a controlled cross-platform benchmark; prompt length, context growth, sampler settings, memory pressure, thermal state, and runtime versions can materially change performance.

Quick start with MLX-VLM

python -m pip install -U mlx-vlm huggingface_hub


python -m mlx_vlm.generate \
  --model Vontra/Qwen3.8-27B-oQ2 \
  --prompt "Explain the difference between linear and full attention." \
  --max-tokens 512

Download for local use:

hf download Vontra/Qwen3.8-27B-oQ2 \
  --local-dir ~/.omlx/models/Vontra/Qwen3.8-27B-oQ2

Using it with oMLX

1. Place the model at ~/.omlx/models/Vontra/Qwen3.8-27B-oQ2.
2. Refresh the oMLX model registry.
3. Load Qwen3.8-27B-oQ2 and use the chat UI or OpenAI-compatible endpoint.

curl "$OMLX_BASE_URL/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OMLX_API_KEY" \
  -d '{
    "model": "Qwen3.8-27B-oQ2",
    "messages": [{"role": "user", "content": "Write a short Swift actor example."}],
    "temperature": 1.0,
    "top_p": 0.95,
    "max_tokens": 256
  }'

For long prompts, begin with a conservative context limit and increase it while watching memory pressure. The configured context is a model capability, not a guarantee that every host can prefill it within available unified memory.

Architecture

Qwen3.8-27B is a dense causal language model with a vision encoder. It uses the Qwen3.5 architectural foundation, interleaving Gated DeltaNet linear-attention blocks with periodic full-attention blocks.

For upstream evaluations, usage guidance, intended use, limitations, safety information, and the full architecture discussion, see the original model card.

Conversion and validation notes

• Source weights: the official Qwen checkpoint.
• Quantization: global 2-bit affine weights with group size 64, with 136 module overrides at 5-bit and one module each at 3-, 6-, and 8-bit.
• The upstream tokenizer, processor files, chat template, and generation configuration are preserved.
• Both quantization and quantization_config preserve the full per-module oQ recipe.
• All 2,180 converted tensors and all three indexed shards were checked locally.
• This is an aggressive low-bit release. Evaluate output quality carefully and prefer oQ4, oQ6, or 8-bit when fidelity matters more than footprint.
• Quantization can reduce output quality relative to the source weights; use a higher-precision variant when quality matters more than memory use.
• The model was loaded and exercised through end-to-end generation on Apple silicon.

This is a community conversion, not an official Qwen release. Validate quality and numerical behaviour on your own representative workload before production use.

Licence and attribution

The upstream model is released under the Apache License 2.0. A copy is included in this repository; review it before use or redistribution.

All model design, training, benchmark, and upstream documentation credit belongs to Qwen and the original contributors. The MLX conversion, Apple-silicon validation, compatibility work, and packaging are provided by Vontra.

Choose for your Mac

64GB Macs · 128GB Macs · 256GB Macs

Published peak memory: 13.44 GB; estimated starting tier: 64GB, leaving about 50 GB nominal headroom. The collections use published M3 Studio peaks with at least 25% nominal headroom; fit on other Macs is an estimate, and full context is not guaranteed. Start with short context and one request.

Runtime and evidence

The exact tested oMLX application version is not recorded here; a library version is not an app version. The original performance tables retain their benchmark conditions and speed figures; this documentation update adds no new test results.

Quick start and demo prompt

hf download Vontra/Qwen3.8-27B-oQ2 --local-dir ./models/Qwen3.8-27B-oQ2

Add the downloaded folder to oMLX model directories, refresh the list, and follow this card's architecture and MTP compatibility requirements before loading.

Try this in a new chat with a 128-token output limit:

Explain why the sky looks blue in three short sentences.

This is a demo prompt to try, not a recorded successful run; a captured demonstration for this documentation update is not yet available.

Follow Vontra for new Apple Silicon releases and fixes.