The first GGUF of Maxwell-V1.0-Flash. We wrote its llama.cpp support. Built from the Maxwell team's original BF16 weights.
maxwell-moe-port.patch in this repo) on llama.cpp d3c3e242c. Stock llama.cpp, LM Studio and Ollama do not load these files yet.Maxwell-V1.0-Flash is a 35B-total / 3B-active Mixture-of-Experts model by the Maxwell team, pre-trained on ~26T multilingual tokens and post-trained for reasoning, search and agentic work, Apache-2.0. Its tokenizer is built for Thai, Lao, Myanmar, Khmer and Shan as much as for CJK and Latin text. This repo has GGUF files of it for llama.cpp.
AD (Atomic Dynamic) files keep everything but the routed experts at Q8_0 (the mHC mixing matrices stay F32) and quantize the experts, which hold ~94% of the weights, with an importance matrix computed on the BF16 weights over our public 4.8M-token calibration build, rendered with Maxwell's own chat template. The name says the expert types: AD-Q5_K-Q4_K has Q5_K down projections and Q4_K gate/up projections. File names are Maxwell-V1-Flash-<name>.gguf; for the split BF16, point llama.cpp at the first part.
| file | GB on disk | bpw | suggested hardware | mean KLD | 99th pct KLD | same top token |
|---|---|---|---|---|---|---|
BF16 (2 parts, folder BF16/) |
72.0 | 16.02 | the original weights, bit-exact | 0 | 0 | 100% |
Q8_0 |
38.4 | 8.54 | 64 GB Macs, 48 GB of GPU memory | 0.0371 | 0.431 | 93.6% |
AD-Q6_K |
30.2 | 6.72 | 48 GB Macs, 2x 24 GB GPUs | 0.0482 | 0.563 | 92.6% |
AD-Q6_K-Q5_K |
27.2 | 6.05 | 32 GB GPUs (RTX 5090) | 0.0697 | 0.826 | 90.9% |
AD-Q5_K-Q4_K |
22.9 | 5.09 | 32 GB with a long context, 32 GB Macs | 0.1383 | 1.649 | 87.3% |
AD-Q4_K-IQ4_XS |
20.8 | 4.62 | 24 GB GPUs (RTX 3090 / 4090) | 0.1736 | 2.045 | 85.7% |
"Same top token" is how often the file picks the same next token as the BF16 original; "mean KLD" is the KL divergence of its next-token distribution from BF16's. Both are measured with llama-perplexity on our 30-language neutral corpus (114 chunks of 4096 tokens), llama.cpp d3c3e242c with the port, 2x H100 80 GB. Read them with the section below: for this model they include a rounding floor of about 0.03. The suggested hardware leaves room for a moderate context.
This model loses more to quantization than most: at 4.6 bits per weight the file picks the original's token 85.7% of the time, where models of this class usually stay above 95%. The plain llama.cpp Q4_K_M preset without an importance matrix (21.9 GB) measures 0.281 mean KLD and 81.6% same top token; AD-Q4_K-IQ4_XS is 1.1 GB smaller with 38% lower mean KLD. Below 20 GB our layouts fell under 80% same top token (18.1 GB: 79.4%, 15.9 GB: 74.8%, 14.0 GB: 68.5%), so we do not publish them. All numbers, logs and the importance matrix are in AtomicChat/Maxwell-V1-Flash-GGUF-metrics.
Speed, one run of llama-bench with flash attention: AD-Q4_K-IQ4_XS on 1x H100 80 GB processes a 512-token prompt at 6,398 tokens/s and generates 173 tokens/s with an empty context; 5,374 and 150 at 8K tokens of context; 3,804 and 132 at 32K. Checked by hand in llama-server --jinja with the same file: a reasoning answer, thinking off, a tool call and an answer in Thai.
Maxwell is unusually sensitive to rounding. The same BF16 weights give a different next-token distribution when llama.cpp multiplies them with bf16-rounded inputs than when it runs everything in F32: mean KLD 0.030, the same top token 93.2% of the time (full model, 3 chunks of 4096 tokens). The KV cache type makes no difference, and no single tensor group is the cause: keeping all attention, the head, the shared and dense FFNs in F32 brings it only to 0.023. Our reading is that small differences anywhere flip which of the 256 experts the router picks near its top-12 boundary, layer after layer. For scale: the vendor's transformers code in bf16 and our BF16 file differ by mean KLD 0.044 on the same tokens.
So the numbers in the table, measured against the BF16 file as for every AtomicChat release, include this floor: even Q8_0 reads 0.037. They still rank the files by how far quantization moves them.
Build llama.cpp at d3c3e242c with the port from this repo:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout d3c3e242c16b8cbf51400fad410cd69b22d64f35
git apply /path/to/maxwell-moe-port.patch
cmake -B build -DGGML_CUDA=ON # Mac: drop -DGGML_CUDA=ON (Metal is on by default)
cmake --build build -j --config Release --target llama-server llama-cli
Then serve a file. The chat template (reasoning, tools) is embedded, so pass --jinja; the sampling below is the vendor's:
./build/bin/llama-server -m Maxwell-V1-Flash-AD-Q4_K-IQ4_XS.gguf --jinja -c 32768 -ngl 99 \
--temp 0.6 --top-p 0.95 --top-k 20
Thinking is on by default. To turn it off for a request, send "chat_template_kwargs": {"enable_thinking": false}, or start the server with --chat-template-kwargs '{"enable_thinking": false}'.
We wrote the maxwell-moe architecture for llama.cpp:
What it does not do yet:
How we checked it against the Maxwell team's modeling_maxwell_moe.py (fp32), with llama.cpp in F32 on the same tokens:
| check (2 chunks of 512 tokens, our neutral corpus) | mean KLD | same top token |
|---|---|---|
| the first block of Maxwell-V1.0-Flash + its head | 0.000002 | 100.0% |
| the first 3 blocks (dense FFN) + head | 0.000002 | 99.6% |
| the first 6 blocks (3 of them MoE) + head | 0.000055 | 100.0% |
| for scale: the vendor's code on CPU against itself on GPU, 6 blocks | 0.000002 | 100.0% |
The small growth with the MoE blocks comes from exact ties in the router: its selection bias is ~20.2, so in fp32 the score differences below ~2e-6 vanish and 1.8% of tokens tie at the top-12 boundary in the first MoE block. PyTorch and ggml break such ties differently.
test-llama-archs -a maxwell-moe passes on CUDA and CPU, and llama-server --jinja parses reasoning and tool calls from the embedded template.
| parameters | 35B total, ~3B active per token (both without embeddings) |
| layers | 45 (3 dense FFN + 42 MoE) |
| hidden / expert FFN | 2048 / 512 (shared expert 1024, dense FFN 6144) |
| experts | 256 routed, 12 active, 1 shared |
| attention | 32 query heads, 4 KV heads, head dim 128; block-sparse in the original |
| context | 256K per the model card (config: 524,288) |
| vocabulary | 200,704 |
| license | Apache-2.0 (LICENSE in this repo, from the base model) |
Quantized by AtomicChat.