Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2-oQ4e
Overview
Qwen3.5-27B is a hybrid attention large language model combining linear attention (GatedDeltaNet) with full softmax attention in a 3:1 ratio. This variant has been fine-tuned on a diverse mixture of code generation, reasoning, and instruction-following datasets, then augmented with a Multi-Token Prediction (MTP) head for speculative decoding.
Upstream model: nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2
Architecture
| Component | Configuration |
|---|---|
| Model type | Qwen3_5ForConditionalGeneration |
| Total layers | 64 |
| Attention pattern | 48 linear attention + 16 full attention (3:1 ratio) |
| Full attention interval | Every 4th layer |
| Hidden size | 5,120 |
| Intermediate size | 17,408 |
| Attention heads | 24 (full attention), 16 key + 48 value (linear attention) |
| Head dimension | 256 |
| Vocab size | 248,320 |
| Max position | 262,144 tokens |
| RoPE | Partial (0.25), θ=10,000,000, M-RoPE interleaved [11,11,10] |
| Activation | SiLU |
| Norm | RMSNorm (ε=1e-6) |
Linear Attention (GatedDeltaNet)
| Parameter | Value |
|---|---|
| Key heads | 16 |
| Value heads | 48 |
| Key head dim | 128 |
| Value head dim | 128 |
| Conv kernel | 4 |
| Attn output gate | Swish |
Multi-Token Prediction (MTP)
| Parameter | Value |
|---|---|
| MTP layers | 1 |
| Dedicated embeddings | No (shared with base model) |
Quantization
| Setting | Value |
|---|---|
| Method | oQ4e (enhanced affine) |
| Default bits | 4 |
| Group size | 64 |
| Mode | Affine |
Mixed-precision layers
lm_head: 8-bit (affine, group_size 64)- Most
linear_attn.in_proj_*,linear_attn.out_proj,mlp.down_proj: 5-bit (affine, group_size 64)
Training Data
This model was fine-tuned on a curated mixture including:
- Coder390: Code generation and completion
- EfficientThink: Efficient reasoning patterns
- Opus5.5: High-quality instruction following
- GPT6Astra: Diverse knowledge and reasoning
- Grok4.7: General conversation and reasoning
- DSV4Pro: Domain-specific tasks
- K3: Knowledge-intensive QA
- SFT: Supervised fine-tuning mixture
- RLOO: Reinforcement learning from policy optimization
Validation
oMLX (Apple Silicon)
- Load: ✅ Model loads successfully
- Tokenizer: ✅ Tokenizer loads and encodes correctly
- Text generation: ✅ Validated via Hermes Agent + oMLX
Status: Model produces coherent output when served via oMLX.
Usage
oMLX (Apple Silicon)
from mlx_lm import load, generate
model, tokenizer = load(
"hermitdave/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2-oQ4e"
)
prompt = "Explain the transformer architecture."
response = generate(model, tokenizer, prompt, max_tokens=512)
print(response)
MLX-LM CLI
mlx_lm.generate \
--model hermitdave/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2-oQ4e \
--prompt "Write a Python function to sort a list." \
--max-tokens 512
Memory Requirements
| Device | Memory |
|---|---|
| M3 Max 64GB | ~16 GB (measured peak) |
| M2 Ultra 128GB | ~16 GB |
| Minimum recommended | 24 GB RAM |
Citation
@misc{qwen38-coder390-mtp-dflash2,
title={Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2},
author={nerkyor},
year={2026},
url={https://huggingface.co/nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2}
}
Attribution
This MLX quantization was prepared by Hermes Agent (Nous Research).