Qwen2.5-32B-Instruct · GGUF Q4_K_M

Evaluation results: this Q4_K_M build compared with F16, capability family by family, with confidence intervals and file size, on the results page: pbhappliedsystems.com/models/qwen2.5-32b-instruct#Q4_K_M

Does Q4_K_M quantization degrade Qwen2.5-32B-Instruct?

Answer (quant_eval v7.22.22): Against its own F16 baseline, Qwen2.5-32B-Instruct at Q4_K_M degraded 0 of 8 capability families and improved 0 (McNemar, p < 0.05; n = 200 per family). No family degraded significantly; the largest movement, toolcall at +1.0 points, is within noise. Full results with intervals

Licence provenance: Apache-2.0 (SPDX Apache-2.0), commercial use permitted_by_recorded_license; upstream Qwen/Qwen2.5-32B-Instruct at revision 5ede1c97bbab6ce5cda5812749b4c0bdf79b18dd (exact); evaluated file SHA-256 6f810a332a884410aa65cc1b5a128a8603f083b36465acfbbf67a08f50a4d3e3.

Other builds of this model: F16 · All evaluated builds (Collection) · Datasets

Contents: Model Description · PBH Applied Systems Evaluation — quant_eval v7.22.22 · Comparability & Fixture Methodology · Paired Comparison: Q4_K_M vs. F16 Baseline · Per-Family Pass Rates — Q4_K_M Variant · Key Findings · Efficiency Gains · Quantization Variants Comparison

Converted and evaluated by PBH Applied Systems, LLC — Applied AI/ML Consulting · LLM Optimization & Deployment · Quantized AI Infrastructure

🔬 This repository is part of a production-oriented evaluation series. Every model published under pbhappliedsystems has been independently evaluated using quant_eval v7.22.21/v7.22.22 — a behavioral evaluation harness developed by PBH Applied Systems. Scores measure real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning families — not perplexity or benchmark leaderboard proxies.

📌 This is the Q4_K_M quantized variant — the recommended choice for production deployment. The full-precision F16 baseline is published separately: F16. Q4_K_M shows no statistically significant difference from F16 in any of the 8 test families (McNemar p < 0.05: 0 of 8) while delivering a 69.7% footprint reduction. This is quantization done right.

---

Try the Live AI Agent Demo & Compare Models

Two complementary interfaces, both free:

1. Quick Demo — Single Model Testing

Launch the PBH Applied Systems AI Agent Demo →

Free · 3 example queries per agent type · No account required · No document upload

Test this model (or others) against three agent workflow templates:

• Reasoning & Analysis — Chain-of-thought decomposition and structured problem-solving
  Example queries: Should a startup build on cloud LLMs or self-host quantized models? · Analyze the trade-offs between model quantization and inference latency. · What are the cost implications of running 32B parameter models on edge devices?

• Document Intelligence — Long-context information extraction and Q&A
  Example queries: Extract key clauses from a contract and summarize legal risks. · Analyze a research paper and identify the novel contributions. · Compare market analysis reports and highlight strategic differences.

• Code & Automation — API scaffolding, data transformation, and task completion
  Example queries: Build a REST API endpoint for user authentication and include rate limiting. · Transform CSV data into a production-ready database schema. · Write a Python script to validate and clean messy customer records.

Every query runs on private GPU infrastructure — your prompts, reasoning traces, and outputs never leave the system and are never sent to Frontier model providers or cloud vendors. This matters for organizations with sensitive data, compliance requirements, or internal knowledge that cannot leave the building.

2. Model Comparison Arena — Side-by-Side Evaluation

Launch the quant_eval Agent Arena Space →

Compare two models in real-time across all three agent types.

The Agent Arena lets you:

• Select any two evaluated models (F16 or quantized variants) and test them side-by-side on the same queries
• View execution traces for both agents, showing chain-of-thought reasoning and tool dispatch
• Check the Model Leaderboard — the Q4_K_M pass rate in each of the eight quant_eval capability families for every model with a published run, marked ▼ degraded or ▲ improved where McNemar's paired test against the model's own full-weight baseline is significant, ordered by how many families degraded
• Read the Methodology tab — explanation of quant_eval, the 8 test families, and how to request a full evaluation report for models not yet in the series

Why compare in the Arena:

The demo shows what this model can do. The Arena shows how this model compares to others. If you're deciding between F16 and Q4_K_M, or between different model sizes, run the same query in the Arena with both selected. You'll see the exact differences in reasoning quality, tool dispatch, and output coherence — not benchmark scores, but real agent behavior.

Evaluation-backed leaderboard: Every score in the Arena comes from published quant_eval results: the quant_eval Public Corpus (DOI 10.5281/zenodo.22009419) and, for showcase models such as Ministral-3-14B-Instruct-2512, sealed run bundles published as summaries. The numbers are not opinionated; they're measured.

---

Model Description

This repository contains the Q4_K_M quantized GGUF of Qwen/Qwen2.5-32B-Instruct, a 32-billion parameter instruction-tuned model developed by Alibaba Qwen Team (2024). Qwen2.5-32B-Instruct features a 32,768-token context window and exceptional multilingual instruction-following capability.

Q4_K_M is a mixed-precision K-quant: mostly 4-bit (Q4_K) weights, with selected tensors kept at 6-bit (Q6_K), delivering a 3.30× compression ratio compared to F16. On this model, Q4_K_M shows no statistically significant difference from F16 in any of the 8 test families — the only Qwen2.5 size in this series where that holds (7B and 14B-1M each show 4 of 8 families differing at p < 0.05).

For nearly all production deployments, Q4_K_M is the right choice. Deploy F16 only in research contexts where cost is irrelevant.

Key Characteristics

• Parameters: 32B
• Format: GGUF Q4_K_M (4-bit mixed-precision)
• File size: 19.85 GB
• SHA256: 6f810a332a884410aa65cc1b5a128a8603f083b36465acfbbf67a08f50a4d3e3
• Context window: 32,768 tokens
• Minimum VRAM (GPU inference): ~21 GB
• Recommended GPU tier: A10G · RTX 6000 · L40 · A100 (40GB)
• Inference speed (eval hardware): avg 16.31 tokens/sec on Modal A10G GPU cluster
• Throughput vs. F16: 0.87× observed on unmatched hardware (Q4_K_M on A10G, F16 on A100-80GB); the ratio reflects an accelerator change, not a precision comparison. On the same hardware as its F16 baseline, Q4_K_M would have smaller inference times (not measured in this run).
• Footprint reduction: 69.7% smaller than F16
• Multilingual: English, Chinese, and other languages

---

PBH Applied Systems Evaluation — quant_eval v7.22.22

Evaluation conducted by PBH Applied Systems, LLC using quant_eval v7.22.22 Run ID: Qwen2.5_32B_Instruct_20260815_081051 · Fixtures: v7.22.22 (SHA256: 0673b5e5...) Evaluation date: August 15, 2026 · Seed: unsupported (Modal); top_k=40, top_p=0.95 · Hardware: Modal GPU cluster · Published: August 15, 2026

Evaluation Data Published to Zenodo

All evaluation artifacts are published under the quant_eval Public Corpus (DOI: 10.5281/zenodo.22009419):

• Run Provenance (D4): 10.5281/zenodo.22010462 — Selected run manifest and rollup fields, artifact SHA-256 hashes, runner and decoding configurations
• Golden Oracle Fixtures (D3): 10.5281/zenodo.22010278 — Test case specifications and fixture version crosswalk
• Family Pass Rates (D6): 10.5281/zenodo.22010623 — Aggregate pass rates per family, both runners, 95% Wilson confidence intervals
• Per-Case Behavioral Results (D1): 10.5281/zenodo.22009799 — 19,200 per-case rows, both precisions, all signals, case-level pass/fail
• Throughput Telemetry (D2): 10.5281/zenodo.22009987 — Per-call generation metrics, wall-time, token counts
• Efficiency & Footprint (D7): 10.5281/zenodo.22010723 — Stored artifact size, compression ratio, tokens/sec, observed wall-time ratios with hardware scope

Evaluation Contract: canonical_agentic_contract_v7.22.18:evaluation

---

Comparability & Fixture Methodology

Fixtures Enable Valid Variant Comparison

The quant_eval evaluation harness uses a consistent fixture set to measure both full-precision and quantized variants. The F16 and Q4_K_M variants were evaluated under the same fixture set, enabling direct comparison.

Fixture Generation:

What this means: Results are directly comparable. Both runners used identical fixtures, enabling precise measurement of Q4_K_M behavioral change relative to F16. In this run, measurement shows zero significant differences across all families.

---

Paired Comparison: Q4_K_M vs. F16 Baseline

Quantization effect on all families — directly measured.

Deltas represent pass-rate change (Q4_K_M − F16), with 95% confidence intervals (paired multinomial, Bonferroni-adjusted Wilson) and paired McNemar p-values for statistical significance. N = 200 cases per family.

* json_multistep pass rates are identical, but 22 cases are discordant (11 F16-only passes, 11 Q4_K_M-only passes). Cluster-adjusted (effective n = 167): +0.012 [−0.070, +0.093].

Summary: Q4_K_M matches F16 case for case on three families (stateful, hybrid, MCQ — 200/200 each) and matches F16's pass rate on multi-step planning. The remaining four show nominal variation (±0.010 or less). No family shows a statistically significant difference (all p ≥ 0.5).

---

Per-Family Pass Rates — Q4_K_M Variant

All rates include 95% Wilson score confidence intervals. N = 200 cases per family.

Source: quant_eval Family Pass Rates, DOI 10.5281/zenodo.22010623

---

Key Findings

Finding 1: Perfect Behavioral Parity — A Rare Achievement

Q4_K_M maintains 100% pass rate on three critical families (stateful, hybrid, MCQ), identical to F16 case for case. Multi-step planning matches F16's rate (0.600 both). In this series, the Qwen2.5-7B and 14B-1M Q4_K_M variants each show significant differences in 4 of 8 families; Qwen2.5-32B under Q4_K_M shows none.

Finding 2: Compression Without Cost

The 3.30× compression ratio (69.7% footprint reduction) comes with no statistically significant behavioral penalty on any family. Every measured delta is nominal (±0.010 or less, p ≥ 0.5).

Finding 3: Hardware Variance, Not Quantization Cost

The observed 0.87× throughput ratio reflects infrastructure differences: F16 was evaluated on Modal A100 GPUs, while Q4_K_M ran on A10G hardware. On the same hardware as its F16 baseline, Q4_K_M would have smaller inference times (not measured in this run). The real story is not speed penalty — it is 69.7% footprint reduction with no statistically significant behavioral loss. For the vast majority of production deployments (batched inference, asynchronous pipelines, cloud GPU billing by VRAM), the memory savings far outweigh any infrastructure-dependent latency variance. Q4_K_M is the obvious choice for cost-efficient deployment.

---

Efficiency Gains

Footprint

Throughput

Hardware caveat: F16 was evaluated on a Modal A100-80GB and Q4_K_M on a Modal A10G. The observed ratios reflect an accelerator change and are not a precision comparison. On the same hardware as its F16 baseline, the quantized model would have smaller inference times.

Interpretation: Q4_K_M delivers a 69.7% footprint reduction with no statistically significant behavioral loss; the observed speed ratios were measured on a smaller GPU and do not represent a quantization slowdown. For memory-constrained production deployments (cloud GPUs billed by VRAM, edge devices, inference batching), Q4_K_M is the clear choice.

---

Quantization Variants Comparison

* Observed on unmatched hardware (F16 on A100-80GB, Q4_K_M on A10G). Use more powerful GPUs for faster inference unless production is memory-constrained or more latency is acceptable.

See the full F16 card:

• F16 — Full-precision baseline; justified only in research contexts

---

Artifact Provenance

The artifact was produced from Qwen/Qwen2.5-32B-Instruct using a custom-built llama.cpp conversion and quantization pipeline developed by PBH Applied Systems.

Artifact verification: SHA256 hashes are recorded in the run manifest (DOI 10.5281/zenodo.22010462). Download the Q4_K_M GGUF and verify:

sha256sum qwen-2.5-32B-instruct-gguf-Q4-K-M.gguf
# Should output: 6f810a332a884410aa65cc1b5a128a8603f083b36465acfbbf67a08f50a4d3e3

---

Evaluation Methodology

quant_eval v7.22.22 is a behavioral evaluation harness developed by PBH Applied Systems. The two-run architecture evaluates the full-precision (F16) model first, then evaluates the quantized variant against the same fixture set, enabling direct measurement of behavioral degradation or improvement.

Fixture set: v7.22.22 evaluation split

• Total unique cases: 1,600 (200 per family × 8 families)
• Per-family test families and pass signals:

Evaluation infrastructure: Modal A10G GPU cluster (nvidia/cuda:12.4.1) Runner: quantized_modal_llama_cpp (llama.cpp via Python binding, version 0.3.20) Decoding: temperature 0.7, top_k=40, top_p=0.95 (seed unsupported on Modal) Context: 8,192-token context window (common Modal cap) Reproducibility: Run manifest, fixtures, and per-case results published to Zenodo under the quant_eval Public Corpus

---

Deployment Recommendations

✅ Deploy Q4_K_M when:

• You want production-grade performance. This is the recommended variant for all production deployments.
• Hardware is constrained. ~21 GB VRAM fits A10G, L40, RTX 6000, and 24 GB consumer GPUs (RTX 3090/4090).
• Cost efficiency drives decisions. 70% footprint reduction directly lowers inference infrastructure cost.
• Behavioral fidelity must be preserved. No statistically significant difference from F16 in any of the 8 test families.
• You need faster inference. On the same hardware as F16, Q4_K_M would be expected to run faster due to reduced memory bandwidth requirements (not measured in this run).

⚠️ Avoid Q4_K_M if:

• You require F16 for research or baseline comparison. The F16 variant serves as the reference standard for behavioral analysis.
• Cost and efficiency are genuinely irrelevant. Even then, Q4_K_M is 69.7% smaller with no statistically significant quality loss.

---

About PBH Applied Systems

PBH Applied Systems, LLC is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development. The organization emphasizes engineering rigor, reproducibility, and real-world deployment constraints — particularly in environments where performance, cost efficiency, and reliability must be balanced against available hardware and budget.

Founder & Principal AI/ML Systems Architect — Patrick Hill

Patrick Hill is the Founder & Principal AI/ML Systems Architect of PBH Applied Systems with 10+ years of experience delivering advanced analytics, predictive modeling, and decision-support solutions across high-stakes operational environments. Patrick holds a Master of Science in Software Engineering with concentrations in Artificial Intelligence and Machine Learning and a B.S. in Business Finance.

Technical expertise spans: Python, SQL, Linux, Pandas, NumPy, scikit-learn, PyTorch, TensorFlow/Keras, HuggingFace Transformers, GGUF, llama.cpp, BitsAndBytes, PEFT, QLoRA, Flask APIs, Docker, CI/CD, Jupyter, Databricks, and quantization strategies.

Published Author: Patrick is the author of Applied Machine Learning: Concepts, Tools, and Case Studies — a 1,200+ page practitioner-oriented textbook adopted as required reading for CSC 373 – Machine Learning at the University of Advancing Technology.

Core Service Areas: LLM optimization & deployment · AI evaluation frameworks · Agentic AI infrastructure · Scalable AI application development · ML pipeline design & analytics · Model & agent cataloging.

---

🔬 About quant_eval & This Evaluation Series

quant_eval is a behavioral evaluation harness developed by PBH Applied Systems, LLC. It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under pbhappliedsystems has been independently evaluated using quant_eval before being recommended for any production role.

See it in action: Live AI Agent Demo →

---

📋 Interested in Deeper Engagement?

This model card documents behavioral evidence under quant_eval. If you're evaluating this model for production deployment, need guidance on quantization impact for your specific workload, or want to understand deployment tradeoffs under your infrastructure constraints, PBH Applied Systems offers several paths:

Async Intake Form — https://pbhappliedsystems.com/contact

Describe your use case, infrastructure, and evaluation needs. Responses are reviewed asynchronously.

Available Services:

• Evaluation Report — A written behavioral audit: per-family pass rates, F16 vs. quantized delta analysis, failure cluster diagnostics, deployment recommendation, and a quantization-level suggestion for your hardware and latency constraints. Evidence-standard numbers only. Typical engagements: $2,500–$5,000.

• Sample report — Read a complete sample report: a real evaluation report for Ministral-3-14B-Instruct-2512, its BF16 baseline against Q4_K_M, Q5_K_M and Q8_0, written exactly as a client receives it.

• Starter-Kit Recommender — Configured agent templates (reasoning, document intelligence, code automation) with proof metrics from your specific workload. Includes model selection, quantization strategy, and deployment architecture guidance tied to your infrastructure.

Submit your details via the form above, and we'll route you to the appropriate engagement.

---

💬 Community

Questions about this model, a model or quant you want measured next, or your own quantization results to compare:

• Discord — PBH Applied Systems: #model-requests, #your-evals, #deployment-help
• GitHub Discussions — quant_eval Public Corpus tools: model requests and polls on what to measure next
• This repo — the Community tab for anything specific to these files

---

License

This GGUF repository inherits the license of the base model: Apache 2.0 — Qwen/Qwen2.5-32B-Instruct

The quant_eval evaluation harness, fuzz prompt builder, and scoring implementation are proprietary to PBH Applied Systems, LLC and are not included in this repository. The golden oracle fixture set used in this evaluation is published under CC BY 4.0 as part of the quant_eval Public Corpus (D3, DOI 10.5281/zenodo.22010278).

---

GGUF quantization and behavioral evaluation performed by PBH Applied Systems, LLC · quant_eval v7.22.22 · Run ID: Qwen2.5_32B_Instruct_20260815_081051 · Published 2026-08-15