GGUF conversions of maxwell-model/Maxwell-V1-Flash (35B total / 3B active, MoE with manifold-constrained hyper-connections and block-sparse attention), converted directly from the released BF16 safetensors. Supports up to 256K context.
🦙 Requires the Maxwell port 🦙
The maxwell-moe architecture is not in stock llama.cpp yet. Build the Maxwell port before loading these files:
git clone --branch maxwell-moe https://github.com/aetherbird/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j --target llama-server llama-cli llama-quantize
Alternatively, apply maxwell-moe-port.patch to llama.cpp master (91979450).
The other published port (AtomicChat) uses dense attention everywhere and drops the MTP block. This port implements the model's block-sparse attention past its 16K dense prefix and exports a usable MTP head, so long-context behaviour matches training.
Generally... Larger files = more precision. Smaller files = more compression = more slop and misbehavin'.
Weights and context share your memory, so be sure to leave headroom. KV costs about 90 KB per token at fp16 (--cache-type-k q8_0 --cache-type-v q8_0 halves it).
| your memory | file | size |
|---|---|---|
| 56 GB+ | Q8_0 / Q8_K_XL | 39.2 / ~42 GB |
| 48 GB | Q6_K | ~31 GB |
| 40 GB | Q5_K_M | ~27 GB |
| 32 GB | Q4_K_M | 22.3 GB |
| 32 GB | IQ4_XS | ~21 GB |
| 24 GB | Q3_K_M | ~17 GB |
| 96 GB+ (or CPU spill) | BF16 (2 parts) | 73.7 GB |
Recommended sampling from the source model card: temperature 0.6, top_p 0.95, top_k 20. Thinking is on by default; disable per request with "chat_template_kwargs": {"enable_thinking": false}. The chat template and tool-call format are embedded, so pass --jinja.
./build/bin/llama-server \
-m Maxwell-V1-Flash-Q4_K_M.gguf \
-c 262144 \
-ngl 99 \
--flash-attn on \
--cache-type-k q8_0 --cache-type-v q8_0 \
--temp 0.6 --top-p 0.95 --top-k 20 \
--jinja
The model attends densely to the first 16K query positions and uses block-sparse attention (8 prefix + 8 local + up to 16 indexer-selected 128-token blocks) beyond that, matching its training. The threshold is a GGUF key (attention.indexer.dense_prefix_tokens).
Flash attention is required for the sparse path; without it the model falls back to dense attention at all lengths.
Every quant bundles the MTP/NextN block. Enable it with --spec-type draft-mtp:
./build/bin/llama-server \
-m Maxwell-V1-Flash-Q8_0.gguf \
-c 262144 -ngl 99 --flash-attn on \
--temp 0.6 --top-p 0.95 --top-k 20 \
--jinja \
--spec-type draft-mtp
Measured on 2xV100 with the Q8_0 file: draft acceptance 0.32, mean accepted length 1.92 tokens per step, not a net speedup on that hardware.
<|maxwell_end|> (id 200001) written as the EOT stop token.maxwell-moe).Generated from the Q8_0 model over a mixed code/English calibration corpus (not the vendor's calibration set):
imatrix/maxwell.imatrix.ggufAll K-quants and i-quants use this matrix. Q8_K_XL keeps F16 embeddings, output head, attention q/k/v/o and indexer projections, with everything else at Q8_0.
test-llama-archs -a maxwell-moe (NMSE, round-trip, mixed batch, KV shift) on CPU and CUDAllama-server --jinja (embedded template)Apache-2.0, inherited from the base model.
maxwell-model/Maxwell-V1-Flash