srswti/axe-superveloce-37b

---

library_name: transformers pipeline_tag: image-text-to-text tags:

• srswti
• bodega
• qwen3_5_moe
• fp8
• quantized
• mixture-of-experts base_model: Qwen/Qwen3.6-35B-A3B model_name: srswti/axe-superveloce-37b license: apache-2.0 description: > axe-superveloce-37B is an architecture-aware compressed variant of Qwen3.6-35B-A3B. Compressed via selective F8_E4M3 compression on linear transformer projections, with routing, recurrent attention, vision, and embedding layers preserved at BF16. Matches or exceeds base model performance on 5 of 8 benchmarks at significantly reduced memory cost.

---

A 37 billion parameter mixture-of-experts model. Built to serve state-of-the-art reasoning, instruction following, and code generation at a fraction of the memory cost of its base model.

The standard approach of quantizing everything uniformly trades correctness for simplicity. We take the opposite position, and a step further: compress aggressively where it is safe to do so, and preserve precision exactly where the architecture is sensitive.

---

This is why F8_E4M3 over simple INT8

INT8 spaces its 256 representable values evenly across the number line. Every step is the same size, regardless of where the weights actually live:

INT8 — uniform spacing, equal steps everywhere

 ← large gap →          ← large gap →
-128    -64    -32   -1  0  1   32    64    128
  |      |      |     |  |  |    |     |      |
  ●      ●      ●     ●  ●  ●    ●     ●      ●

F8_E4M3 spends its representable values where neural network weights actually cluster -- densely packed near zero, thinning out toward the extremes:

F8_E4M3 — non-uniform spacing, dense near zero

          many fine steps here
               ↓↓↓↓↓↓↓
-448  -4  -1  -0.25  0  0.25  1  4  448
  |    |   |    ||||  |  ||||  |  |    |
  ●    ●   ●  ●●●●●●  ●  ●●●●●●  ●    ●
               ↑↑↑↑↑↑
          most weights live here

The result: F8_E4M3 represents typical weight distributions with smaller rounding error than INT8 at the same bit width -- which is why FP8 compression consistently loses less accuracy than INT8 compression.

Say you have a weight with value 0.0317. You need to store it in 8 bits.

With INT8, your available grid points near zero are:

... -2/128   -1/128    0    1/128   2/128 ...
    -0.0156  -0.0078   0   0.0078  0.0156

The closest representable value to 0.0317 is 0.0313 -- off by 0.0004. Fine.

But now say another weight is 0.0021. The closest INT8 value is still 0.0078 -- off by 0.0057. That is a 270% relative error on a small weight, because the grid steps are too coarse near zero.

With F8_E4M3, the grid near zero has many more tick marks packed in. You can represent 0.0021 much more faithfully because the format was designed knowing that most weights live exactly there.

When you round a weight to the nearest grid point, that rounding error flows forward through every matrix multiply that weight participates in. If most of your weights have small rounding errors, the accumulated output error across a 96-layer model stays small. INT8's uniform grid burns precision on ranges the model never uses. F8_E4M3 concentrates precision exactly where the model needs it.

At its core, this architecture is meant to serve higher batch serving

The core computation in every linear layer is a matrix multiply. For an input activation matrix $X$ and a weight matrix $W$, the output is:

$$Y = XW^T$$

In BF16, both $X$ and $W$ are 16-bit values. The multiply-accumulate happens in FP32 accumulator registers on the GPU, and the output is written back in BF16. Memory bandwidth cost per element: 2 bytes for weights, 2 bytes for activations.

In F8_E4M3, the same operation runs differently. Weights are stored as 8-bit values including the language model heads. Before the matrix multiply, each channel is rescaled by a learned per-channel scale factor $s_c$ so that the full dynamic range of that channel's weights maps onto the F8_E4M3 grid as efficiently as possible:

$$W_{quant} = \text{round}_{E4M3}\left(\frac{W}{s_c}\right)$$

At inference, the multiply-accumulate runs on compressed weights, and the result is rescaled back:

$$Y = (X \cdot W_{quant}^T) \times s_c$$

For activations, the scale is not precomputed. It is derived token by token at runtime. For each token vector $x_t$, the scale is:

$$s_t = \frac{\max(|x_t|)}{448}$$

The activation is quantised, the matrix multiply executes, and the result is dequantized before passing to the next operation. This all happens within the same kernel. From the outside, it is invisible.

What this means for throughput. GPU memory bandwidth is the primary bottleneck for autoregressive inference. At BF16, loading a weight matrix costs 2 bytes per parameter. At F8_E4M3, it costs 1 byte. The matrix multiply itself runs on the same tensor cores, but the time spent moving data from VRAM to compute units is halved. For large batch serving where compute is the bottleneck, modern GPUs also expose native F8 tensor core paths with higher theoretical throughput than BF16.

---

Precision Mapping Across the Architecture

Through our own layer-by-layer profiling of activation distributions, routing sensitivity, and accumulated rounding error across the full architecture, we identified exactly which components can absorb 8-bit compression without behavioral change.

Quantized to F8_E4M3

All standard linear projections within the transformer blocks: Q, K, V, and output projections in attention, and the up, gate, and down projections in the routed expert MLPs. These layers represent the overwhelming majority of parameter count and memory bandwidth in the model.

Preserved at BF16

---

Memory and KV Cache

Every quantized weight drops from 2 bytes to 1 byte. For the layers that are quantized, this is a direct 2x reduction in the memory required to hold the model.

The KV cache savings compound on top. During inference, every processed token writes a key vector and a value vector into a cache that persists for the duration of the request. The size of that cache is:

$$\text{KV Cache} = 2 \times L \times H \times d \times T \times b$$

Where $L$ is the number of layers, $H$ is the number of KV heads, $d$ is the head dimension, $T$ is the sequence length, and $b$ is bytes per element. Halving $b$ from 2 (BF16) to 1 (F8) halves the KV cache at every sequence length. At 32K tokens, this frees several gigabytes per active request. That headroom goes directly toward concurrent capacity. Same hardware, more users.

---

Benchmarks

Base model: Qwen/Qwen3.6-35B-A3B. All evaluations run at 0-shot using lm-evaluation-harness and lighteval, served with vLLM under --language-model-only.

On five of eight benchmarks, Axe Superveloce matches or exceeds the base model score. The compressed model outperforms its uncompressed counterpart on coding, a result consistent with per-channel weight scaling producing a tighter effective dynamic range on the neurons most active during code generation tasks.

---

Deployment via vLLM

Axe Superveloce is fully compatible with vLLM and loads natively without additional configuration.

Text only -- skip the vision encoder to free VRAM for additional KV cache:

vllm serve srswti/axe-superveloce-37b --reasoning-parser qwen3 --language-model-only

Multimodal -- full vision and language support:

vllm serve srswti/axe-superveloce-37b --reasoning-parser qwen3

Tool use:

vllm serve srswti/axe-superveloce-37b --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

Speculative decoding via Multi-Token Prediction:

vllm serve srswti/axe-superveloce-37b --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'

Send requests using the OpenAI-compatible endpoint:

from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://<your-server-host>:8000/v1",
)

messages = [
    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]

response = client.chat.completions.create(
    model="srswti/axe-superveloce-37b",
    messages=messages,
)

print(response.choices[0].message.content)

Developed by SRSWTI Inc. - Building world's fastest retrieval and inference engines.