Mistral-Nemo-Instruct-2407 · GGUF Q5_K_M

Evaluation results: this Q5_K_M build compared with F16, capability family by family, with confidence intervals and file size, on the results page: pbhappliedsystems.com/models/mistral-nemo-instruct-2407#Q5_K_M

Does Q5_K_M quantization degrade Mistral-Nemo-Instruct-2407?

Answer (quant_eval v7.22.22): Against its own F16 baseline, Mistral-Nemo-Instruct-2407 at Q5_K_M degraded 0 of 8 capability families and improved 0 (McNemar, p < 0.05; n = 200 per family). No family degraded significantly; the largest movement, mcq at −1.0 points, is within noise. Full results with intervals

Licence provenance: Apache-2.0 (SPDX Apache-2.0), commercial use permitted_by_recorded_license; upstream mistralai/Mistral-Nemo-Instruct-2407 at revision 8aedd450f2583e9c67fae1929f6936b8fc5aef9c (exact); evaluated file SHA-256 20df665d95c88cd3ab144a0153dc977b3c308ae48e42abcefe63891b502e31f1.

Other builds of this model: F16 · Q4_K_M · Q8_0 · All evaluated builds (Collection) · Datasets

Contents: Model Description · PBH Applied Systems Evaluation — quant_eval v7.22.22 · Comparability & Fixture Methodology · Per-Family Paired Comparison — F16 Baseline vs. Q5_K_M · Key Findings · Efficiency & Footprint · Quantization Variants Comparison · Artifact Provenance

Quantized, converted, and evaluated by PBH Applied Systems, LLC — Applied AI/ML Consulting · LLM Optimization & Deployment · Quantized AI Infrastructure

🔬 This repository is part of a production-oriented evaluation series. Every model published under pbhappliedsystems has been independently evaluated using quant_eval v7.22.21/v7.22.22 — a behavioral evaluation harness developed by PBH Applied Systems. Scores measure real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning families — not perplexity or benchmark leaderboard proxies.

✓ Minimal degradation from F16 to Q5_K_M. No family shows a statistically significant difference from F16 (McNemar p < 0.05: 0 of 8). Stateful execution (1.000), schema-only tool calls (0.995), and hybrid responses (1.000) hold at or near F16 levels — the three families where Q4_K_M shows its largest losses. Q5_K_M achieves a 64% footprint reduction and 2.65× generation throughput vs F16 on the same hardware.

---

Try the Live AI Agent Demo & Compare Models

Two complementary interfaces, both free:

1. Quick Demo — Single Model Testing

Launch the PBH Applied Systems AI Agent Demo →

Free · 3 example queries per agent type · No account required · No document upload

Test this model (or others) against three agent workflow templates:

• Reasoning & Analysis — Chain-of-thought decomposition and structured problem-solving
  Example queries: Should a startup build on cloud LLMs or self-host quantized models? · Analyze the trade-offs between model quantization and inference latency. · What are the cost implications of running 14B parameter models on edge devices?

• Document Intelligence — Long-context information extraction and Q&A
  Example queries: Extract key clauses from a contract and summarize legal risks. · Analyze a research paper and identify the novel contributions. · Compare market analysis reports and highlight strategic differences.

• Code & Automation — API scaffolding, data transformation, and task completion
  Example queries: Build a REST API endpoint for user authentication and include rate limiting. · Transform CSV data into a production-ready database schema. · Write a Python script to validate and clean messy customer records.

Every query runs on private GPU infrastructure — your prompts, reasoning traces, and outputs never leave the system and are never sent to Frontier model providers or cloud vendors. This matters for organizations with sensitive data, compliance requirements, or internal knowledge that cannot leave the building.

2. Model Comparison Arena — Side-by-Side Evaluation

Launch the quant_eval Agent Arena Space →

Compare two models in real-time across all three agent types.

The Agent Arena lets you:

• Select any two evaluated models (F16 or quantized variants) and test them side-by-side on the same queries
• View execution traces for both agents, showing chain-of-thought reasoning and tool dispatch
• Check the Model Leaderboard — the Q4_K_M pass rate in each of the eight quant_eval capability families for every model with a published run, marked ▼ degraded or ▲ improved where McNemar's paired test against the model's own full-weight baseline is significant, ordered by how many families degraded
• Read the Methodology tab — explanation of quant_eval, the 8 test families, and how to request a full evaluation report for models not yet in the series

Why compare in the Arena:

The demo shows what this model can do. The Arena shows how this model compares to others. If you're deciding between F16 and Q5_K_M, or between this model and another in the series, run the same query in the Arena with both selected. You'll see the exact differences in reasoning quality, tool dispatch, and output coherence — not benchmark scores, but real agent behavior.

Evaluation-backed leaderboard: Every score in the Arena comes from published quant_eval results: the quant_eval Public Corpus (DOI 10.5281/zenodo.22009419) and, for showcase models such as Ministral-3-14B-Instruct-2512, sealed run bundles published as summaries. The numbers are not opinionated; they're measured.

---

Model Description

This repository contains the 5-bit quantized (Q5_K_M) GGUF of mistralai/Mistral-Nemo-Instruct-2407, a 12-billion parameter instruction-tuned model developed by Mistral AI in collaboration with NVIDIA (July 2024 release). Mistral-Nemo features the Tekken tokenizer and a 128,000-token context window — the largest in the PBH Applied Systems evaluated series outside of Qwen2.5-14B-Instruct-1M, which supports a 1 million token window.

The Q5_K_M format applies 5-bit quantization with K-quant medium precision. As documented in the evaluation section below, Q5_K_M demonstrates minimal behavioral degradation while providing 64% footprint reduction and 2.65× generation throughput compared to F16 on the same hardware. This quantization is the recommended balanced variant — the sweet spot between compression and reliability.

The full-precision F16 baseline is published separately at pbhappliedsystems/mistral-nemo-instruct-2407-gguf-F16.

Key Characteristics

• Parameters: 12B
• Format: GGUF Q5_K_M
• File size: 8.73 GB
• SHA256: 20df665d95c88cd3ab144a0153dc977b3c308ae48e42abcefe63891b502e31f1
• Context window: 128,000 tokens (Tekken tokenizer)
• Minimum VRAM (GPU inference): ~11 GB (standard T4 or better)
• Recommended GPU tier: NVIDIA T4 (16 GB) · RTX 3070/4070 · A10G
• Inference speed (eval hardware): avg 79.97 tokens/sec on NVIDIA RTX 4090
• Speedup vs F16 (same run, same hardware): 2.65× generation throughput (30.14 → 79.97 tokens/sec); 2.56× observed evaluation wall-time ratio
• Multilingual: English, French, German, Spanish, Italian, Portuguese, Russian, Chinese, Japanese

---

PBH Applied Systems Evaluation — quant_eval v7.22.22

Evaluation conducted by PBH Applied Systems, LLC using quant_eval v7.22.22 Run ID: Mistral_Nemo_Instruct_2407_20260815_113254 · Fixtures: v7.22.22 (SHA256: 0673b5e...) Evaluation date: August 15, 2026 · Seed: 42 · Hardware: NVIDIA RTX 4090 · Published: August 15, 2026

Evaluation Data Published to Zenodo

All evaluation artifacts are published under the quant_eval Public Corpus (DOI: 10.5281/zenodo.22009419):

• Paired Degradation Statistics (D5): 10.5281/zenodo.22010557 — McNemar tests, Wilson intervals, cluster-adjusted paired deltas across all evaluated precisions
• Run Provenance (D4): 10.5281/zenodo.22010462 — Selected run manifest and rollup fields, artifact SHA-256 hashes, runner and decoding configurations
• Golden Oracle Fixtures (D3): 10.5281/zenodo.22010278 — Test case specifications and fixture version crosswalk
• Per-Case Behavioral Results (D1): 10.5281/zenodo.22009799 — 19,200 per-case rows, both precisions, all signals, case-level pass/fail
• Family Pass Rates (D6): 10.5281/zenodo.22010623 — Aggregate pass rates per family, both runners, 95% Wilson confidence intervals
• Throughput Telemetry (D2): 10.5281/zenodo.22009987 — Per-call generation metrics, wall-time, token counts
• Efficiency & Footprint (D7): 10.5281/zenodo.22010723 — Stored artifact size, compression ratio, tokens/sec, observed wall-time ratios with hardware scope

Evaluation Contract: canonical_agentic_contract_v7.22.18:evaluation

---

Comparability & Fixture Methodology

Identical Fixtures Across All Three Runs

All three Mistral-Nemo runs were evaluated against the same 1,600 cases. A full structural comparison of the fixture files across bundles shows the only differing key is the top-level version label: all cases, oracle expectations, and trace_sha256 values are identical, and fixture_schema_version is 7.22.13 in every bundle.

Baseline reproducibility: The F16 baseline was executed in all three runs, on different days and under both contract revisions. All eight family pass rates are identical across the three runs, and at case level 0 of 1,600 cases differ in any pairwise comparison. The Q4_K_M, Q5_K_M, and Q8_0 deltas are all measured against the same baseline.

The Zenodo datasets record the fixture version-label crosswalk (D3) and per-run manifests (D4) for full reproducibility.

---

Per-Family Paired Comparison — F16 Baseline vs. Q5_K_M

All rates include 95% Wilson score confidence intervals. McNemar p-values test statistical significance of paired differences. N = 200 cases per family.

Sources:

• quant_eval Family Pass Rates, DOI 10.5281/zenodo.22010623
• quant_eval Paired Degradation Statistics, DOI 10.5281/zenodo.22010557

---

Key Findings

Finding 1: Stateful and Hybrid Execution — Fully Preserved

Q5_K_M holds F16-level performance on the three agent families where Q4_K_M shows its largest losses.

Why it matters: These three families measure the core agent-execution layer — maintaining state, dispatching tools, and emitting structured responses. Q5_K_M preserves all three at F16 levels. This is a significant difference from Q4_K_M, which degrades on all three (−0.170 stateful, −0.280 hybrid, −0.215 tool).

Implication: Q5_K_M does not require the additional scaffolding Q4_K_M needs. Stateful, schema-only tool-call, and hybrid-response behavior match F16 within measurement noise — at roughly one-third of F16's file size.

Finding 2: Multi-Choice and Tool Dispatch — Negligible Loss

Two families show minimal, non-significant degradation:

Why it matters: The McNemar p-values (0.500, 0.625) indicate that differences are not statistically significant. The observed deltas are within the noise band. These families are operationally equivalent between F16 and Q5_K_M.

Implication: Workflows depending on tool execution or multi-choice extraction show no statistically significant quantization effect under Q5_K_M.

Finding 3: Structured Output — Consistent Baseline Preserved

Both json and json_multistep families are unchanged within measurement noise:

Why it matters: These low F16 pass rates are a model characteristic, not a quantization artifact. Q5_K_M does not amplify the model's structural-output struggles; it preserves them at F16 levels. If you need higher structured-output fidelity, this is a model limitation independent of quantization level.

---

Efficiency & Footprint

Source: quant_eval Efficiency & Footprint, DOI 10.5281/zenodo.22010723

Q5_K_M is the efficiency winner: larger footprint reduction than Q8_0 (64% vs 47%), faster than Q8_0 (2.65× vs 1.78× same-run generation throughput), slightly larger than Q4_K_M (8.73 GB vs 7.48 GB), while avoiding Q4_K_M's behavioral costs. For GPU-constrained deployments, this is the quantization to choose.

---

Quantization Variants Comparison

See how this model performs across all three quantized variants. All three were evaluated on identical fixtures against the same F16 baseline.

How to read this table: Speed is the observed evaluation wall-time ratio (F16 ÷ quantized) on matched RTX 4090 hardware — not a controlled throughput benchmark. Stateful/Toolcall-only/Hybrid are pass rates.

See the full cards:

• F16 — Baseline reference
• Q4_K_M — Extreme compression, significant behavioral cost
• Q8_0 — Conservative quantization, high fidelity, modest speedup

---

Artifact Provenance

Both artifacts were produced from mistralai/Mistral-Nemo-Instruct-2407 using a custom-built llama.cpp conversion and quantization pipeline developed by PBH Applied Systems.

Artifact verification: SHA256 hashes are recorded in the run manifest (DOI 10.5281/zenodo.22010462). Download the Q5_K_M GGUF and verify:

sha256sum Mistral_Nemo_Instruct_2407_Q5_K_M.gguf
# Should output: 20df665d95c88cd3ab144a0153dc977b3c308ae48e42abcefe63891b502e31f1

---

Evaluation Methodology

quant_eval v7.22.22 is a behavioral evaluation harness developed by PBH Applied Systems. The two-run architecture evaluates both the full-precision (F16) and quantized variant against an identical fixture set, enabling direct comparison of behavioral degradation.

Fixture set: v7.22.22 evaluation split

• SHA256: 0673b5e5c0fb1b7ba4bdc8c9aa985104260db183453c5708aff14c87823edb97
• Total unique cases: 1,600 (200 per family × 8 families)

Evaluation hardware: NVIDIA RTX 4090 (24 GB VRAM) Runner: quantized_llama_cpp (llama.cpp via Python binding, version 0.3.16) Decoding: temperature 0.3 (per Mistral AI's published recommendation), seed 42, top_p 1.0 Context: 4,096-token context window (llama.cpp configuration) Reproducibility: Run manifest, fixtures, per-case results, and paired statistics published to Zenodo under the quant_eval Public Corpus

---

Deployment Recommendations

✅ Deploy Q5_K_M when:

• You need a balanced quantization. Minimal degradation across all families; reliable for production agent workflows.
• GPU memory is constrained to ~11–16 GB. Q5_K_M fits on T4 (16 GB) and RTX 3070/4070 tiers.
• Latency matters but reliability matters more. 2.65× generation throughput vs F16 on the same hardware, with no statistically significant degradation in any family.
• Your workflow includes stateful conversations. Q5_K_M maintains state perfectly; no scaffolding needed.
• Tool calling and hybrid responses are non-negotiable. Q5_K_M preserves both at F16 levels.
• You want to avoid Q4_K_M's guardrail complexity. No additional retry or fallback logic beyond what F16 itself requires.

⚠️ Avoid Q5_K_M if:

• You need the reference baseline itself. If you have ~24.5 GB VRAM and require exact F16 behavior (e.g., for audit or reproduction), use F16. In this evaluation Q5_K_M showed no statistically significant difference from F16 in any family.
• You need absolute minimal footprint. Q4_K_M is 14% smaller (7.48 GB vs 8.73 GB), if you can tolerate its behavioral costs.

🎯 Recommended Deployment:

Q5_K_M is the default quantization for this model. Deploy it as your primary inference variant with the same validation you would apply to F16 — it needs none of the additional guardrails Q4_K_M requires. It delivers:

• ✓ Agent execution reliability (stateful, tool-calling, hybrid responses)
• ✓ Substantial efficiency gains (64% footprint reduction, 2.65× generation throughput)
• ✓ Broad hardware compatibility (~11 GB VRAM)
• ✓ Minimal operational complexity

---

About PBH Applied Systems

PBH Applied Systems, LLC is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development. The organization emphasizes engineering rigor, reproducibility, and real-world deployment constraints — particularly in environments where performance, cost efficiency, and reliability must be balanced against available hardware and budget.

Founder & Principal AI/ML Systems Architect — Patrick Hill

Patrick Hill is the Founder & Principal AI/ML Systems Architect of PBH Applied Systems with 10+ years of experience delivering advanced analytics, predictive modeling, and decision-support solutions across high-stakes operational environments. Patrick holds a Master of Science in Software Engineering with concentrations in Artificial Intelligence and Machine Learning and a B.S. in Business Finance.

Technical expertise spans: Python, SQL, Linux, Pandas, NumPy, scikit-learn, PyTorch, TensorFlow/Keras, HuggingFace Transformers, GGUF, llama.cpp, BitsAndBytes, PEFT, QLoRA, Flask APIs, Docker, CI/CD, Jupyter, Databricks, and quantization strategies.

Published Author: Patrick is the author of Applied Machine Learning: Concepts, Tools, and Case Studies — a 1,200+ page practitioner-oriented textbook adopted as required reading for CSC 373 – Machine Learning at the University of Advancing Technology.

Core Service Areas: LLM optimization & deployment · AI evaluation frameworks · Agentic AI infrastructure · Scalable AI application development · ML pipeline design & analytics · Model & agent cataloging.

---

🔬 About quant_eval & This Evaluation Series

quant_eval is a behavioral evaluation harness developed by PBH Applied Systems, LLC. It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under pbhappliedsystems has been independently evaluated using quant_eval before being recommended for any production role.

See it in action: Live AI Agent Demo →

---

📋 Interested in Deeper Engagement?

This model card documents behavioral evidence under quant_eval. If you're evaluating this model for production deployment, need guidance on quantization impact for your specific workload, or want to understand deployment tradeoffs under your infrastructure constraints, PBH Applied Systems offers several paths:

Async Intake Form — https://pbhappliedsystems.com/contact

Describe your use case, infrastructure, and evaluation needs. Responses are reviewed asynchronously.

Available Services:

• Evaluation Report — A written behavioral audit: per-family pass rates, F16 vs. quantized delta analysis, failure cluster diagnostics, deployment recommendation, and a quantization-level suggestion for your hardware and latency constraints. Evidence-standard numbers only. Typical engagements: $2,500–$5,000.

• Sample report — Read a complete sample report: a real evaluation report for Ministral-3-14B-Instruct-2512, its BF16 baseline against Q4_K_M, Q5_K_M and Q8_0, written exactly as a client receives it.

• Starter-Kit Recommender — Configured agent templates (reasoning, document intelligence, code automation) with proof metrics from your specific workload. Includes model selection, quantization strategy, and deployment architecture guidance tied to your infrastructure.

Submit your details via the form above, and we'll route you to the appropriate engagement.

---

💬 Community

Questions about this model, a model or quant you want measured next, or your own quantization results to compare:

• Discord — PBH Applied Systems: #model-requests, #your-evals, #deployment-help
• GitHub Discussions — quant_eval Public Corpus tools: model requests and polls on what to measure next
• This repo — the Community tab for anything specific to these files

---

License

This GGUF repository inherits the license of the base model: Apache 2.0 — mistralai/Mistral-Nemo-Instruct-2407

The quant_eval evaluation harness, fuzz prompt builder, and scoring implementation are proprietary to PBH Applied Systems, LLC and are not included in this repository. The golden oracle fixture set used in this evaluation is published under CC BY 4.0 as part of the quant_eval Public Corpus (D3, DOI 10.5281/zenodo.22010278).

---

GGUF conversion, quantization, and behavioral evaluation performed by PBH Applied Systems, LLC · quant_eval v7.22.22 · Run ID: Mistral_Nemo_Instruct_2407_20260815_113254 · Published 2026-08-15