Sign up

pbhappliedsystems/qwen-2.5-32B-instruct-gguf-F16

🤗 On Hugging Faceapache-2.066 GBGGUF✓ HF checksums availableupdated today
Magnet

Qwen2.5-32B-Instruct · GGUF F16

Converted by PBH Applied Systems, LLC

— Applied AI/ML Consulting · LLM Optimization & Deployment · Quantized AI Infrastructure

📌 Provenance repository — no behavioral evaluation performed. This repository contains the full-precision F16 GGUF of Qwen2.5-32B-Instruct. At 65.5 GB, the F16 artifact exceeds the VRAM capacity of the evaluation hardware (NVIDIA RTX 4090, 24 GB). All behavioral evaluation data for this model is in the Q4\_K\_M companion repository: pbhappliedsystems/qwen-2.5-32B-instruct-gguf-Q4-K-M.
🔬 About the evaluation series. Every other model published under pbhappliedsystems has been independently evaluated using quant_eval v7.21 — a proprietary behavioral evaluation harness developed by PBH Applied Systems — at both F16 and Q4\_K\_M precision. For Qwen2.5-32B, the F16 GGUF was produced and its artifact provenance is recorded here, but the evaluation constraint is documented honestly rather than omitted.

Why No Evaluation

In the PBH Applied Systems evaluation pipeline, F16 GGUFs serve as cache-generation baselines for Q4\_K\_M comparison runs. For every other model in the series, the F16 run produces a full_weight_cache.json that the Q4\_K\_M run reuses, enabling a direct two-run comparison against identical fixtures.

For Qwen2.5-32B, the F16 GGUF is 65.5 GB. Loading this into the evaluation hardware (NVIDIA RTX 4090, 24 GB VRAM) is not possible — not even with partial CPU offload at the precision required for a valid cache-generation baseline. The Q4\_K\_M comparison run (20260221_144732) was therefore run as a standalone evaluation against freshly generated responses rather than against a cached F16 baseline.

The consequence for this repository: There is no full_weight_cache.json, no F16 evaluation CSV, and no cross-precision comparison data. This card exists to document the F16 artifact for provenance and to make the 65.5 GB GGUF accessible to users with appropriate hardware.

For full behavioral analysis, cross-series comparisons, and deployment recommendations, see the Q4\_K\_M card.

Try the Live AI Agent Demo

Launch the PBH Applied Systems AI Agent Demo →

This model is part of the PBH Applied Systems evaluated model series that supports the live AI Agent Demo. The demo lets visitors interact with production-style agent workflows powered by open-weight language models evaluated through PBH Applied Systems' quant_eval framework.

The F16 model serves a different role than the Q4_K_M deployment variant. F16 is the full-precision baseline used to measure what the model can do before quantization. quant_eval then compares the quantized model against this baseline to identify which capabilities are preserved, which degrade, and which tasks require guardrails or a higher-precision deployment.

This comparison is central to the demo. It helps determine which model belongs in which agent role:

  • Reasoning models are selected for planning, analysis, and auditable decision workflows.
  • Document models are selected for long-context extraction, summarization, and structured Q&A.
  • Code models are selected for task completion, structured output, API scaffolding, and automation workflows.
  • Quantized variants are selected when they preserve enough behavior to reduce cost, latency, and GPU requirements.
  • F16 variants remain important when maximum fidelity, cleaner tool execution, or reduced quantization risk matters more than speed or cost.

The live demo shows the deployment side of that process. The F16 card documents the reference behavior. The Q4_K_M card shows what changes after compression. Together, they explain how PBH Applied Systems uses quant_eval to choose the correct LLM for the correct agent type instead of guessing from model size or leaderboard reputation.

Model Description

This repository contains the full-precision F16 GGUF of Qwen/Qwen2.5-32B-Instruct, a 32.5-billion parameter instruction-tuned model from Alibaba Cloud (September 2024 release).

Key Characteristics

  • Parameters: 32.5B (31.0B non-embedding)
  • Architecture: Causal LM · RoPE · SwiGLU · RMSNorm · Attention QKV bias · GQA (40 Q heads / 8 KV heads)
  • Layers: 64
  • Format: GGUF F16 (full precision)
  • File size: 65.5 GB
  • SHA256: 02e264f0273624b39b0650f8c0583c6d04c320c777780ca5be839999912adf3c
  • Context window: 131,072 tokens (full); 8,192 tokens (generation)
  • Default config targets 32,768 tokens; YaRN scaling required beyond that — see Long Context section
  • Minimum VRAM (full GPU offload): ~80 GB
  • Recommended hardware: 2× A100 80 GB · 4× A100 40 GB · 3× A10G 24 GB
  • License: Apache 2.0

Artifact Provenance

| Artifact | Format | Size | SHA256 | Evaluated |

|---|---|---|---|---|

| qwen-2.5-32B-instruct-gguf-F16.gguf | GGUF F16 | 65.5 GB | 02e264f0273624b39b0650f8c0583c6d04c320c777780ca5be839999912adf3c | ❌ VRAM constraint |

| Q4\_K\_M (companion repo) | GGUF Q4\_K\_M | 19.9 GB | 6f810a332a884410aa65cc1b5a128a8603f083b36465acfbbf67a08f50a4d3e3 | ✅ Run 20260221_144732 |

The F16 GGUF was converted from Qwen/Qwen2.5-32B-Instruct using a custom-built llama.cpp conversion pipeline developed by PBH Applied Systems, without modification to model weights.

quant_eval context: F16 run 20260221_144732 was requested with quant_types: ['Q4_K_M']. Due to VRAM constraints, no F16 full_weight_cache.json was written. The Q4\_K\_M evaluation ran as a standalone quantized_llama_cpp runner evaluation.

Hardware Requirements

| Configuration | VRAM Required | Notes |

|---|---|---|

| F16 (this repo) · full GPU | ~80 GB | 2× A100 80 GB minimum |

| F16 · 4-GPU split | ~20 GB per GPU | 4× A100 40 GB or 4× A10G 24 GB |

| F16 · partial CPU offload | ~40–50 GB VRAM + 64 GB RAM | Reduced context; slower inference |

| Q4\_K\_M (companion repo) | ~24 GB | Single A10G or RTX 4090 |

For most production use cases, the Q4\_K\_M variant is the correct choice. It runs on single-GPU hardware, is fully evaluated, and delivers the same structured behavioral outputs that F16 would produce on this class of tasks. F16 is appropriate for compliance environments requiring full-weight artifacts, research settings requiring exact weight fidelity, or future evaluation runs on multi-GPU infrastructure.

Long-Context Deployment

The default config.json targets 32,768 tokens. To enable the full 131,072-token context window, apply YaRN scaling by adding the following to config.json before conversion, or configure it at the llama.cpp level:

{
  "rope_scaling": {
    "factor": 4.0,
    "original_max_position_embeddings": 32768,
    "type": "yarn"
  }
}

Note that YaRN in llama.cpp uses static scaling — the factor is constant regardless of actual input length. Apply only when processing long contexts is required; shorter-context inference may be marginally affected.

Usage

Installation

pip install llama-cpp-python huggingface_hub

For multi-GPU CUDA deployment:

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir

Python — llama-cpp-python (multi-GPU)

from huggingface_hub import hf_hub_download
from llama_cpp import Llama

# Note: 65.5 GB download — requires ~80 GB total VRAM for full GPU offload
model_path = hf_hub_download(
    repo_id="pbhappliedsystems/qwen-2.5-32B-instruct-gguf-F16",
    filename="qwen-2.5-32B-instruct-gguf-F16.gguf"
)

# Multi-GPU: tensor_split distributes model layers across available GPUs
# Example: 2× A100 80 GB
llm = Llama(
    model_path=model_path,
    n_ctx=8192,
    n_gpu_layers=-1,       # Offload all layers
    tensor_split=[1, 1],   # Equal split across 2 GPUs
    verbose=True,          # Monitor GPU memory allocation
)

response = llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": "You are a precise assistant. Follow instructions exactly."
        },
        {
            "role": "user",
            "content": "Analyze the following and return a JSON object with keys: summary, risk_level, action_items."
        }
    ],
    temperature=0.7,
    max_tokens=1024,
)
print(response["choices"][0]["message"]["content"])

For partial CPU offload when full VRAM is unavailable (slower, but functional):

# Example: 2× A10G 24 GB (48 GB total VRAM) + 64 GB system RAM
llm = Llama(
    model_path=model_path,
    n_ctx=4096,
    n_gpu_layers=48,       # Offload first N layers; tune based on available VRAM
    tensor_split=[1, 1],
    verbose=True,
)

CLI — llama-cli (multi-GPU)

llama-cli \
  --model qwen-2.5-32B-instruct-gguf-F16.gguf \
  --chat-template qwen2 \
  --system-prompt "You are a precise assistant. Follow instructions exactly." \
  --prompt "Return a JSON object with keys: summary, risk_level, action_items." \
  --n-predict 1024 \
  --ctx-size 8192 \
  --n-gpu-layers -1 \
  --tensor-split 1,1 \
  --temp 0.7

For server deployment:

llama-server \
  --model qwen-2.5-32B-instruct-gguf-F16.gguf \
  --chat-template qwen2 \
  --ctx-size 8192 \
  --n-gpu-layers -1 \
  --tensor-split 1,1 \
  --port 8080 \
  --host 0.0.0.0

🔬 About quant_eval & This Evaluation Series

quant_eval is a proprietary behavioral evaluation harness developed by PBH Applied Systems, LLC. It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under pbhappliedsystems has been independently evaluated using quant_eval before being recommended for any production role.

See it in action: Live AI Agent Demo →

The demo runs production-style agent workflows powered by open-weight models selected through the quant_eval evaluation pipeline.

Need a deployment recommendation?
Not sure which quantization level is right for your hardware, latency target, or agent type?
→ pbhappliedsystems.com

Evaluated and published by PBH Applied Systems, LLC · patrick@pbhappliedsystems.com

About PBH Applied Systems

PBH Applied Systems, LLC is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development.

Patrick Hill, M.S. — Founder · Data Scientist · AI/ML Engineer · Author of Applied Machine Learning: Concepts, Tools, and Case Studies (required reading, UAT CSC 373)

Core Service Areas: LLM Optimization & Deployment · AI Evaluation Frameworks · Agentic AI Infrastructure · Scalable AI Application Development · ML Pipeline Design & Analytics · Model & Agent Cataloging

📞 Work With PBH Applied Systems

👉 Book a Scoping Call · 👉 Request an Evaluation Report — from $2,500

Connect

| | |

|---|---|

| 🌐 | pbhappliedsystems.com |

| 📧 | patrick@pbhappliedsystems.com |

| 💼 | LinkedIn |

| ▶️ | YouTube |

| 📸 | Instagram |

| 👍 | Facebook |

License

This GGUF repository inherits the license of the base model:

Apache 2.0 — Qwen/Qwen2.5-32B-Instruct

The quant_eval evaluation methodology and fixture set are proprietary to PBH Applied Systems, LLC and are not included in this repository.

GGUF conversion performed by PBH Applied Systems, LLC · No behavioral evaluation — see companion Q4\_K\_M repository for all evaluation data