HERETIC 1.4.0MTPAPEXMIT

Ornith-1.5-35B-A3B-Heretic-MTP-APEX

Heretic Trial 62 BF16 merge · replaced MTP head · APEX GGUFs + mmproj

🐦 About this release

Ornith-1.5-35B-A3B is DeepReinforce's Qwen3.5 MoE agentic coding model. This experimental derivative merges the selected Heretic 1.4.0 Trial 62 LoRA into BF16 weights.

The shipped native MTP head is untrained (initializer-like statistics). It is replaced here with the 19-tensor fused BF16 head from shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY (Qwen3.6 graft + 12K KL distillation). Refusal-modified models can behave differently from the original; evaluate before deployment. License: MIT.

🧠 Model Details

Architecture	Qwen3.5 MoE, multimodal
Parameters	35B total, 3B active per token · 256 experts, 8 active
Layers	40 transformer layers + 1 fused MTP layer (19 tensors, Q8_0 in APEX)
Context	262,144 tokens
License	MIT

⚡ Heretic Trial 62 of 80

Classic Heretic (MPOA, rank-3, o_proj + down_proj). Per-layer refusal directions. Search log: 9/100 refusals @ first-token KL 0.0107. Re-score after export: 11/100 @ KL 0.0105.

direction_index	per layer
attn.o_proj	max 3.19 · pos 29.14 · min 2.89 · dist 14.92
mlp.down_proj	max 3.43 · pos 23.58 · min 1.62 · dist 8.45

🔁 MTP head

Official Ornith-1.5 mtp.* tensors match random init (e.g. q_proj std 0.01993 / kurtosis 2.997; mtp.norm mean 0.02281). Native draft acceptance is poor. This release removes those 785 unpacked tensors and grafts shisa's 19 fused tensors (sha256 73c6e839…de712e). APEX keeps the whole MTP block at Q8_0.

🚀 Usage

llama.cpp (vision + text, recent build with Qwen3.5 MoE MTP)

hf download SC117/Ornith-1.5-35B-A3B-Heretic-MTP-APEX-GGUF --include "*.gguf" --local-dir ./models ./llama-server -m ./models/Ornith-1.5-35B-A3B-Heretic-MTP-APEX-I-Compact.gguf --mmproj ./models/mmproj-Ornith-1.5-35B-A3B-Heretic-MTP-BF16.gguf -ngl 99 -c 131072 --draft-mtp

🎛️ Recommended Settings

General / Coding

temperature=0.6, top_p=0.95, top_k=20, do_sample=true

💡 What is APEX?

These GGUFs use APEX, an MoE-aware mixed-precision method. Routed experts compress hardest, shared experts stay high, attention follows a layer-wise gradient. The MTP layer remains Q8_0. I-variants use the original Ornith-1.5 trunk imatrix (40 layers; MTP has no imatrix rows and does not need them at Q8_0). The matching vision projector remains BF16.

📦 APEX Quantization Tiers

File	Size	Best For
`*-APEX-I-Quality.gguf`	23.49 GB	Highest quality
`*-APEX-I-Balanced.gguf`	26.06 GB	Best all-rounder
`*-APEX-I-Compact.gguf`	17.33 GB	Best quality/size ratio
`*-APEX-I-Mini.gguf`	14.27 GB	Most compact

Sizes are measured from the final files.

❓ FAQ: Why is I-Balanced larger than I-Quality?

Short answer: the tiers are bit-allocation strategies, not a size ladder. APEX assigns precision per tensor from measured importance, so file size does not grow monotonically with the tier name.

I-Quality keeps every sensitive tensor high-precision — attention Q6_K in all 40 blocks, shared experts Q8_0, edge blocks and the MTP layer high as well — but compresses the redundant middle routed experts (blk.10–30) aggressively to IQ4_XS. Routed experts carry most of an MoE's parameters, and the importance analysis shows the middle blocks absorb this compression with virtually no measurable loss. That is where the bytes are saved.

I-Balanced is the conservative, uniform profile: no tensor anywhere below Q5_K. Uniformity simply costs more bytes.

So bigger does not mean better. I-Quality is the smarter bit allocation and remains the highest-quality preset despite the smaller size; if you need a smaller file, step down to I-Compact or I-Mini instead of picking by file size.

Tensor group	I-Quality	I-Balanced
Routed experts · edge blocks (0–4, 35–39)	Q6_K	Q6_K
Routed experts · blk.5–9, 31–34	Q5_K	Q5_K
Routed experts · middle blocks (10–30)	IQ4_XS	Q5_K
Shared experts · all blocks	Q8_0	Q8_0
Attention · all 40 blocks	Q6_K	Q6_K
MTP layer (blk.40)	Q8_0	Q8_0

The two tiers are identical everywhere except the middle routed-expert band — that single band is the whole size difference.

Citation

@misc{ornith-1.5-35b,
    title = {{Ornith-1.5-35B-A3B}: From Self-Scaffolding to Self-Improvement},
    url = {https://ornith.ai/ornith_1_5.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}

SC117/Ornith-1.5-35B-A3B-Heretic-MTP-APEX-GGUF

Ornith-1.5-35B-A3B-Heretic-MTP-APEX

Links

Citation