Apex-35B-V1.0-FP8
FP8 (W8A8, block-wise) quantization of Apexintelligence-AI/Apex-35B-V1.0 in the compressed-tensors format, produced with llm-compressor.
- ✅ Vision encoder included: image (and video) input works. The 27-layer ViT tower from the source checkpoint is kept in BF16.
- ✅ MTP head included: Apex's own multi-token-prediction head, kept in BF16, for native speculative decoding in SGLang and vLLM.
- ✅ 262K-token context (native, unchanged from the source model).
- ✅ ~38 GB on disk (vs ~72 GB in BF16). Fits on 2× 48 GB GPUs (L40S / A6000 Ada / RTX 6000 Ada) or one 80 GB GPU.
Model Overview
Apex-35B-V1.0 is the first model from Apex Intelligence. It is a general-purpose and research model built on the Qwen3.6-35B-A3B architecture, aimed at coding, tool use, question answering, deep search, AI research, computational science and robotics simulation. It uses dynamic thinking. See the original model card for details and benchmarks.
| Property | Value |
|---|---|
| Architecture | Qwen3_5MoeForConditionalGeneration |
| Parameters | 35B total, ~3B activated |
| Layers | 40 (30 Gated DeltaNet linear-attention + 10 gated full-attention, interval 4) |
| Experts | 256 routed (8 active) + 1 shared; expert intermediate size 512 |
| Hidden size | 2048 |
| Attention | 16 Q heads / 2 KV heads, head dim 256, partial RoPE 0.25 |
| Vision encoder | 27-layer ViT, patch 16, spatial merge 2 |
| MTP | 1 layer |
| Vocabulary | 248,320 |
| Context length | 262,144 |
Quantization Details
| Method | llm-compressor oneshot, QuantizationModifier(scheme="FP8_BLOCK") |
| Weights | FP8 E4M3, symmetric, static 128×128 block scales |
| Activations | FP8 E4M3, symmetric, dynamic per-token-group (1×128) |
| Calibration data | none (data-free min-max) |
| Quantized | all attention and linear-attention projections, shared experts, all 10,240 routed expert MLPs |
| Kept in BF16 | vision tower, embed_tokens, lm_head, MoE router (mlp.gate), shared_expert_gate, linear_attn.in_proj_a/b, norms, conv / SSM parameters, the entire MTP head |
| Format | compressed-tensors (float-quantized), weights stored per expert |
These layer choices match Qwen's official Qwen3.6-35B-A3B-FP8 release.
About the vision tower and MTP head
Both come straight from the Apex checkpoint and were not modified:
- Vision: the
model.visual.*weights are present in the source upload. They are kept in BF16 and were verified end-to-end (OCR and natural-photo description, see Verification below). - MTP: the source's
mtp.*weights are stored inmodel_mtp.safetensorsin BF16 and excluded from quantization throughre:.*mtp\..*in theignorelist, to maximize draft acceptance.
Recommended Parameters
The sampling defaults are in generation_config.json, and SGLang and vLLM apply them automatically. These are the values recommended by Apex Intelligence:
| Parameter | Value |
|---|---|
temperature |
1.0 |
top_p |
0.95 |
top_k |
20 |
max_tokens |
65,536 |
The model thinks by default (<think>…</think>). Its chat template (chat_template.jinja) is Apex's own and is kept byte-identical; do not override it with a generic Qwen template. Use the qwen3 reasoning parser to separate the reasoning from the answer, and the qwen3_coder tool-call parser for tool use.
Hardware Requirements
| Setup | Notes |
|---|---|
| 2× 48 GB (L40S / A6000 Ada / RTX 6000 Ada), TP=2 | ~19 GB of weights per GPU; plenty of KV cache for long context |
| 1× 80 GB (H100 / A100 80G*) | single-GPU serving |
| FP8 compute | native on Ada (sm89), Hopper (sm90) and Blackwell. *On Ampere, vLLM falls back to weight-only FP8 (Marlin). |
Serving
SGLang (tested with v0.5.23)
python3 -m sglang.launch_server \
--model-path ramachandrajoshi/Apex-35B-V1.0-FP8 \
--served-model-name apex-35b-fp8 \
--tp 2 \
--context-length 262144 \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--host 0.0.0.0 --port 30000
To turn off MTP, remove the --speculative-* flags.
vLLM
vllm serve ramachandrajoshi/Apex-35B-V1.0-FP8 \
--served-model-name apex-35b-fp8 \
--tensor-parallel-size 2 \
--max-model-len 262144 \
--speculative-config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}' \
--reasoning-parser qwen3 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder \
--enable-prefix-caching
Usage (OpenAI-compatible API)
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
# Text / coding
resp = client.chat.completions.create(
model="apex-35b-fp8",
messages=[{"role": "user", "content": "Write a Python function that merges overlapping intervals."}],
temperature=1.0, top_p=0.95, extra_body={"top_k": 20},
max_tokens=65536,
)
print(resp.choices[0].message.content)
# Vision
resp = client.chat.completions.create(
model="apex-35b-fp8",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/3/3a/Cat03.jpg"}},
{"type": "text", "text": "Describe this image."},
]}],
max_tokens=4096,
)
print(resp.choices[0].message.content)
Verification
Served with SGLang 0.5.23 on 2× NVIDIA L40S (TP=2):
| Check | Result |
|---|---|
| Load | ✅ CompressedTensorsW8A8Fp8MoE for experts, block-FP8 linear for dense layers; 17.2 GB of weights per GPU plus 1.7 GB of BF16 MTP |
| Text reasoning | ✅ correct (17*23 + 144/12 → 403) |
| Code generation | ✅ correct |
| Vision: OCR (synthetic image) | ✅ exact text (INVOICE #4729), plus correct shapes and colours |
| Vision: natural photo | ✅ accurate description (orange tabby cat, green eyes, red hose in background) |
| MTP speculative decoding (3 steps, 4 draft tokens) | ✅ average accept length 3.1–3.4 of 4 (≈69–80% acceptance) |
| Single-stream decode speed | ~245 tok/s |
| Weight fidelity | ~2.7% relative dequantization error per tensor (normal FP8 E4M3 rounding) |
No full benchmark suite was run against the BF16 model. Block-wise FP8 is usually near-lossless for this model family.
Reproduction
The scripts are in quantization/, and the recipe is in recipe.yaml:
quantize.pyruns llm-compressoroneshotwithFP8_BLOCK(data-free) on the original checkpoint.fix_mtp.pyverifies the expert split, stores the MTP experts per expert (same layout as the backbone) and addsre:.*mtp\..*to the ignore list.
Tooling: llm-compressor 0.15.0 · compressed-tensors 0.20.1 · transformers 5.21.0 · torch 2.14.1.
License & Credits
Released under Apache-2.0, the same license as the source model. All credit for the model goes to Apex Intelligence and the Qwen team. This is an unofficial community quantization. See the original model card for intended use, limitations and safety considerations.