Groot-19B-A3B-261023-MTP-GGUF

Groot-19B-A3B-261023 is a compact, open-weight (Apache-2.0), natively multimodal Mixture-of-Experts (MoE) reasoning model built for complex real-world tasks, agentic coding, and native tool use. Derived from Qwen3.6-35B-A3B and post-trained on fewer than 0.1 billion open-source tokens, it comprises roughly 19B total parameters (plus a 0.4B Multi-Token Prediction layer) with only ~3B activated per token across 40 hybrid layers (10 full-attention and 30 linear-attention) and 128 routed experts. Despite using 45.7% fewer parameters than Qwen3.6-35B-A3B and 26.9% fewer than Gemma-4-26B-A4B, it outperforms both on 12 of 16 benchmarks (averaging 41.44 vs. 38.08 and 30.05) while consuming 23.1% and 65.6% fewer output tokens, respectively. The model operates in thinking mode by default—with configurable non-thinking and preserved-thinking modes—and supports a native 262,144-token (256K) context window extensible to roughly 1M tokens via YaRN. Released in BF16, FP8, NVFP4, and GGUF (Q4_K_M) weights for deployment on vLLM, SGLang, KTransformers, and Hugging Face Transformers, it is engineered for accessible hardware, running a 128K context on a 16GB RTX 4060 Ti and the full 256K context on 24GB consumer GPUs (RTX 3090/4090), an RTX 5090, or 128GB unified-memory systems.

[!NOTE] MTP — Speculative Decoding: Yes

[!NOTE] Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.

Model Files

BF16 Test Run

Tested Groot-19B-A3B-261023.BF16.gguf with vision support on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB VRAM, using llama-cli with full GPU offloading and MTP speculative decoding.

Performance Metrics

Test Shots

Run Inference

llama-cli \
    -m /path/to/models/Groot-19B-A3B-261023.BF16.gguf \
    --spec-type draft-mtp \
    --spec-draft-n-max 2 \
    -ngl 99 \
    -c 16384 \
    --cache-type-k q8_0 \
    --seed 42 \
    -n 8192 \
    -st \
    -p "Write a program in C to insert a new node at the end of a Singly Linked List."

Performance results reflect this specific hardware and inference configuration. Actual throughput may vary depending on the prompt, output length, model, software build, and runtime settings.

llama.cpp

LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp

---

Snapshots (5)

---

Quantized using llama.cpp by prithivMLmods