Agents-A1 GGUF Quants

High quality GGUF quantizations of InternScience/Agents-A1, a 35B Qwen3.5-MoE agent model.

These files were produced from the BF16 Hugging Face checkpoint with a patched llama.cpp build that supports the qwen35moe architecture. The calibration pass used an importance matrix built from coding/instruction chat data, then each quant was benchmarked against the BF16 GGUF reference.

Recommended Files

Files

Metrics

Hardware and runtime profile:

• GPU: single NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, full offload
• llama.cpp flags: -ngl 99 -sm none -fa on -p 512 -n 128 -b 4096 -ub 512 -r 3
• PPL: llama-perplexity, context 2048, 64 rendered eval conversations, 3 chunks
• KLD: approximate KL(P_BF16 || P_quant) over top-64 next-token distributions on 32 prompts

The PPL eval is intentionally small, so treat PPL deltas as directional. KLD and top-1 agreement are more useful here for quant-to-BF16 comparison.

Charts

Raw metric files are in metrics/; KLD reports, checksums, and the MTP audit are in reports/.

MTP Variants

The upstream Agents-A1 checkpoint used for the first GGUF release advertises MTP in config but does not ship mtp.*/blk.40.* tensors. The MTP variants here graft in the Agents-A1 MTPLX MTP sidecar from wang-yang/Agents-A1-MTPLX-Q4. The dense MTP block is preserved at Q6_K/F32 while the model body remains quantized to IQ4_XS, Q4_K_M, or Q5_K_M.

The Q5_K_M MTP artifact reuses the validated Q5_K_M body tensors and the same Q6_K/F32 blk.40.* MTP block, with GGUF metadata updated to qwen35moe.block_count=41 and qwen35moe.nextn_predict_layers=1.

Structural checks for the integrated MTP GGUFs:

Single-user serving profile: one RTX PRO 6000 Blackwell Max-Q 96 GB GPU, PARALLEL=1, CTX_SIZE=8192, streaming chat completions, 12 requests, 128 max tokens, temperature=0, top_p=1.

agents-a1-Q5_K_M-MTP-graft-headQ6.gguf is structurally validated but has not been re-profiled for throughput or draft acceptance yet.

Recommended low-latency/single-user throughput profile: SPEC_DRAFT_N_MAX=2. Recommended high-acceptance fallback: SPEC_DRAFT_N_MAX=1.

Detailed MTP evidence is in:

• reports/agents-a1-mtp-q4-profile-summary.md
• reports/agents-a1-mtp-q4-profile-summary.json
• reports/agents-a1-q5-mtp-build-report.json
• configs/mtp_profiles.yaml

Usage

Example with the recommended compact quant:

llama-server \
  -m agents-a1-IQ4_XS.gguf \
  -ngl 99 \
  -c 8192 \
  -b 4096 \
  -ub 512 \
  --flash-attn on

NVFP4 example:

llama-server \
  -m agents-a1-NVFP4.gguf \
  -ngl 99 \
  -c 8192 \
  -b 4096 \
  -ub 512 \
  --flash-attn on

The NVFP4 artifact is a standard GGUF using the NVFP4 tensor type, but runtime support is still newer and less universal than K-quants or IQ4_XS. It was tested on a Blackwell GPU with a llama.cpp build reporting BLACKWELL_NATIVE_FP4 = 1.

MTP example:

LLAMA_SPEC_MAX_DRAFTING_SLOTS=1 \
LLAMA_MTP_FAST_BACKEND_SAMPLE=1 \
LLAMA_MTP_DRAFT_TOP_K=1 \
LLAMA_MTP_DRAFT_TOP_P=1 \
LLAMA_MTP_DRAFT_TEMP=1 \
llama-server \
  -m agents-a1-IQ4_XS-MTP-graft-headQ6.gguf \
  -ngl 99 \
  -c 8192 \
  -b 4096 \
  -ub 512 \
  --flash-attn on \
  --reasoning off \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --spec-draft-n-min 0 \
  --spec-draft-backend-sampling

For the high-acceptance profile, change --spec-draft-n-max 2 to --spec-draft-n-max 1. The same command shape applies to agents-a1-Q4_K_M-MTP-graft-headQ6.gguf and agents-a1-Q5_K_M-MTP-graft-headQ6.gguf.

Vision / mmproj

The release includes one shared multimodal projector:

• mmproj-agents-a1-bf16.gguf
• processor_config.json
• preprocessor_config.json
• video_preprocessor_config.json

The mmproj was converted from the original InternScience/Agents-A1 Hugging Face checkpoint with llama.cpp convert_hf_to_gguf.py --mmproj --outtype bf16. It contains the Qwen3VL vision tower/projector and is independent of the text quantization level, so the same file is intended for Q4-class and higher text GGUFs:

• agents-a1-IQ4_XS.gguf
• agents-a1-IQ4_XS-MTP-graft-headQ6.gguf
• agents-a1-NVFP4.gguf
• agents-a1-Q4_K_M.gguf
• agents-a1-Q4_K_M-MTP-graft-headQ6.gguf
• agents-a1-Q5_K_M.gguf
• agents-a1-Q5_K_M-MTP-graft-headQ6.gguf
• agents-a1-Q6_K.gguf
• agents-a1-Q8_0.gguf

Q3_K_M may load with the same mmproj, but it is not the recommended vision profile because image tasks are more sensitive to text-model quantization.

Example with llama.cpp's multimodal CLI:

llama-mtmd-cli \
  -m agents-a1-Q4_K_M.gguf \
  --mmproj mmproj-agents-a1-bf16.gguf \
  --image image.jpg \
  -p "Describe the image." \
  -ngl 99 \
  -c 4096 \
  -b 1024 \
  -ub 256 \
  --chat-template chatml \
  --image-min-tokens 1024 \
  --flash-attn on

If your llama.cpp llama-server build has multimodal support enabled, the same mmproj can be passed with --mmproj mmproj-agents-a1-bf16.gguf.

Local smoke test:

Verification report: reports/mmproj-q4km-actual-image-verify.json.

MTP Status

The original upstream snapshot remains config-only for MTP; see reports/mtp-weights-audit.json. The new *-MTP-graft-headQ6.gguf files are true integrated MTP GGUFs built from the Agents-A1 MTPLX MTP sidecar.

Provenance

• Base model: InternScience/Agents-A1
• License: Apache-2.0, inherited from the base model
• Quantization source: BF16 GGUF converted from the Hugging Face checkpoint
• MTP source: wang-yang/Agents-A1-MTPLX-Q4 sidecar grafted onto the base Agents-A1 checkpoint
• Calibration: coding/instruction chat data rendered with the model chat template
• Quantizer: patched llama.cpp with Qwen3.5-MoE and NVFP4 support