Ornith-1.5-35B-A3B-Heretic-GB10 (NVFP4/FP8 Mixed Quantization)

[!IMPORTANT] Derivative Work Notice. This is an unofficial community conversion of a third-party model. The underlying weights are derived from OS-Software/Ornith-1.5-35B-A3B-heretic-ja, a decensored version of ornith-ai/Ornith-1.5-35B-A3B created with Heretic v1.4.0+custom using the Arbitrary-Rank Ablation (ARA) method (LoRA adapter + row-norm preservation). All rights to the original models remain with their creators. This conversion is provided AS-IS under the MIT license, with no warranty of any kind. No original model weights are claimed as our own. "Ornith" and the original model are the work of ornith-ai; the Heretic-JA abliteration is the work of OS-Software.

Content warning: Abliterated / uncensored models have had safety-refusal behavior removed. Outputs may include content that is harmful, offensive, or unlawful. The maintainers are not responsible for generated content. Use responsibly and in accordance with your local laws and the original models' usage terms.

Overview

A mixed-precision NVFP4/FP8 conversion of the Ornith-1.5-35B-A3B Heretic-JA model, optimized for NVIDIA GB10 (DGX Spark / Spark Mini, SM121) — a GPU without native FP4 compute. Decode on GB10 is memory-bandwidth-bound, so 4-bit expert weights cut bytes moved per token by ~3× and run through fused Marlin/compressed-tensors kernels — a large speedup over BF16 even without native FP4 tensor cores.

Key properties:

• Full VLM preserved — vision tower kept in BF16 (image + video input works)
• MTP head kept — speculative decoding via MTP supported (vLLM --speculative-config)
• Reasoning model — use --reasoning-parser qwen3 (output arrives in the reasoning field)
• compressed-tensors format — loads in vLLM ≥ 0.26 out of the box

Quantization Recipe

Total size: ~21 GB. Same recipe family as the official ornith-ai/Ornith-1.5-35B-A3B-NVFP4, with W4A4 (rather than W4A16) on experts.

Benchmarks (measured)

Single GB10 (121 GB unified memory, vLLM 0.26.1rc1, SM121):

• Protocol: /v1/completions, greedy, ignore_eos, 800 tokens, 1 warmup, median of 3
• Context: 262,144 native. KV cache is small (hybrid attention: only 10/40 layers are full-attention)
• Chat sanity: thinking arrives in reasoning field; answers clean

Serving (vLLM)

vllm serve /path/to/Ornith-1.5-35B-A3B-Heretic-GB10 \
  --served-model-name Ornith-35b \
  --port 8000 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.85 \
  --reasoning-parser qwen3 \
  --safetensors-load-strategy=prefetch \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Known limitations

• NVFP4 kernels run via Marlin weight-only path on SM121 unless FlashInfer is built from source (FLASHINFER_CUDA_ARCH_LIST="12.0f 12.1a"); with FlashInfer, dense FP8 layers pick up FlashInferFP8ScaledMM kernels
• Abliteration may reduce performance on refusal-adjacent benchmarks; no eval scores claimed
• Calibrated quantization: dense layers use dynamic FP8 scales (calibration-free); expert scales from the source conversion pipeline

Credits

• ornith-ai — the original Ornith-1.5-35B-A3B model (MIT)
• OS-Software — the Heretic-JA decensored model this is converted from (Heretic v1.4.0+custom, ARA method)
• Heretic/abliteration methodology community