Qwen2.5-32B-Instruct · BitsAndBytes 4-bit (NF4)

Quantized and evaluated by PBH Applied Systems, LLC — Applied AI/ML Consulting · LLM Optimization & Deployment · Quantized AI Infrastructure

🔬 This repository is part of a production-oriented evaluation series. Every model published under pbhappliedsystems has been independently evaluated using quant_eval v7.21 — a proprietary behavioral evaluation harness developed by PBH Applied Systems. Scores measure real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning families — not perplexity or leaderboard proxies.

⚠️ Single-runner evaluation. No FP16 baseline was evaluated for this model. The FP16 Qwen2.5-32B-Instruct (~65 GB) exceeds the VRAM capacity of the evaluation hardware (NVIDIA RTX 4090, 24 GB). All behavioral data in this card comes from the 4-bit quantized_bitsandbytes runner only.

🧩 Backend note. This is the BitsAndBytes NF4 variant — an HF-format safetensors checkpoint loaded directly through Transformers (or served via vLLM). It is a distinct backend from the GGUF Q4_K_M sibling. Where the two diverge in evaluation, it is called out explicitly below.

---

Try This Model in the Live AI Agent Demo

Launch the PBH Applied Systems AI Agent Demo →

This model is part of the PBH Applied Systems live AI Agent Demo, where visitors can test evaluated quantized open-weight models across production-style agent workflows: reasoning and analysis, document intelligence, and code automation.

The demo uses quant_eval results to show how model selection changes by task. A model that performs well for stateful multi-turn workflows or tool dispatch may not be the best choice for hard multi-step planning. Each deployed model is evaluated for practical agent behavior, including coherence, instruction following, reasoning, task completion, structured output reliability, tool-use behavior, and quantization impact.

For this repository, the NF4 4-bit variant represents a deployment-focused checkpoint: it fits on a single 24 GB GPU, loads directly with Transformers, and — unlike the GGUF path — emits clean output with no end-of-sequence token contamination. The evaluation results below explain where this quantized model preserves useful behavior, where 4-bit quantization introduces risk, and what guardrails are recommended before production deployment.

---

Model Description

This repository contains the BitsAndBytes 4-bit (NF4, double-quantized) checkpoint of Qwen/Qwen2.5-32B-Instruct, a 32-billion parameter instruction-tuned model from Alibaba Cloud. The 4-bit weights are stored in HF safetensors with the quantization configuration baked into config.json, so the model loads directly through Transformers with no separate quantization step.

Key Characteristics

• Parameters: 32B (Qwen2ForCausalLM)
• Format: BitsAndBytes 4-bit · NF4 · double-quant
• Compute dtype: float16
• Quant storage: uint8
• File size: ~17.9 GB (4 × safetensors shards + quant_config.json)
• Minimum VRAM (GPU inference): ~22–24 GB
• Recommended GPU tier: RTX 4090 24 GB · A10G 24 GB · A100 40 GB
• Context window: 32,768 tokens
• Inference speed (eval hardware): avg 4.405 sec/case on RTX 4090
• License: Apache 2.0
• FP16 equivalent: ~65 GB — exceeds single-GPU 24 GB VRAM; requires multi-GPU or large-memory server

Quantization Configuration

{
  "quant_method": "bitsandbytes",
  "load_in_4bit": true,
  "bnb_4bit_quant_type": "nf4",
  "bnb_4bit_use_double_quant": true,
  "bnb_4bit_compute_dtype": "float16",
  "bnb_4bit_quant_storage": "uint8"
}

---

PBH Applied Systems Evaluation — quant_eval v7.21

Evaluation conducted by PBH Applied Systems, LLC using quant_eval v7.21 Run ID: 20260629_002808 · Fixtures: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c...) · Seed: 42 Hardware: NVIDIA RTX 4090 · Runner: quantized_bitsandbytes (4-bit only) · Total rows: 42

Per-Family Pass Rates (NF4 4-bit)

---

Key Findings

Finding 1: Multi-Step Planning — Premature STOP (0.400)

json_multistep is the one family where this 4-bit checkpoint shows a genuine limitation. The format is always valid (schema_ok 1.000, stop_semantics_ok 1.000), but the model consistently under-plans, emitting STOP one step early or selecting the wrong placement on the hard case.

Case-level breakdown:

On ms_easy_02 and ms_med_02 the model reasons consistently (checks_consistent_ok=1) but stops before completing the plan. On ms_hard_01 the intermediate reasoning is itself inconsistent (checks_consistent_ok=0) and the final plan is wrong.

What this means for production: do not depend on this model for unassisted multi-step planning. Wrap it with an external planner/oracle validation loop, or select a model with a stronger json_multistep profile if planning reliability is the primary requirement.

Finding 2: MCQ — Perfect 5/5

Clean single-letter extraction, no A-bias, at 0.21 sec/case.

Finding 3: toolcall — Correct and Clean (No EOS Contamination)

Both toolcall cases pass stage-1 (valid dispatch) and stage-2 (correct final value) for a perfect bucket score of 11.000:

This is a notable backend advantage: the GGUF Q4_K_M sibling passes stage-1 but fails stage-2 with final_mismatch caused by trailing <|im_end|> tokens. The Transformers/BnB path decodes with skip_special_tokens=True, so the final answer arrives clean — no EOS stripping required downstream.

Finding 4: toolcall_only — Correct Values, Wrapper Normalized (1.000)

The model emits the correct argument values but a non-standard outer wrapper:

The argument value names ("a", "b") and values are correct; the model uses "tool"/"params" where the strict schema expects "tool_name"/"args". quant_eval v7.21 normalizes the common wrapper aliases (args / arguments / params / parameters) for both scoring and tool execution, so args_ok=1.000. If your downstream stack does not normalize wrappers, enforce the exact outer keys via system prompt (see Usage).

Finding 5: Stateful and Hybrid — Clean at 4-bit

stateful_followup (1.000) and mixed_brief_json (1.000) both pass cleanly, again with no EOS contamination:

---

Backend Comparison — NF4 4-bit vs GGUF Q4_K_M (same 32B)

The two 4-bit backends are close on structured JSON, stateful, hybrid, and MCQ. The NF4 path is cleaner for tool-calling output (no EOS suffix) and faster per case in this run, while the GGUF Q4_K_M path is slightly stronger on multi-step planning. Choose NF4 when you want direct Transformers/vLLM serving and clean tool-call output; choose GGUF Q4_K_M when planning reliability is the deciding factor.

---

Signal-Level Diagnostics (NF4 4-bit)

json_multistep

toolcall_only

stateful_followup / mixed_brief_json

All gating signals (turn1/2_parse_ok, turn1/2_exact_match; answer_line_ok, json_parse_ok, schema_ok) = 1.000.

---

Recommended Use Cases

✅ Deploy with Confidence (NF4 4-bit)

• Tool-calling agents — toolcall 11.000 (clean), toolcall_only 1.000. Best-in-series output cleanliness: no EOS stripping required.
• Stateful multi-turn agents — 1.000 at both turns, clean JSON state.
• Hybrid brief + JSON responses — mixed_brief_json 1.000 at ~2.5 sec/case.
• Structured JSON outputs (single-step) — json and fuzz both bucket=10.000.
• MCQ / single-choice extraction — 5/5 perfect at 0.21 sec/case.
• Complex generation tasks — 32B parameter capacity for nuanced, long-form, or multilingual outputs.

⚠️ Use with Guardrails (NF4 4-bit)

• Multi-step planning — 0.400 pass rate; consistent premature-STOP. Add an external planner/oracle validation loop, or prefer the GGUF Q4_K_M variant (0.600) if planning is the primary requirement.
• Bare tool-call dispatch in non-normalizing stacks — the model emits tool/params wrappers. quant_eval normalizes these; if your runtime does not, enforce exact outer keys (tool_name/args) in the system prompt. Argument value names (a/b) are already correct.

---

Hardware Requirements

---

Usage

Installation

pip install "transformers>=4.46" "bitsandbytes>=0.46.1" accelerate huggingface_hub

BitsAndBytes 4-bit inference requires a CUDA GPU.

Python — Transformers (loads the baked-in 4-bit config)

from transformers import AutoTokenizer, AutoModelForCausalLM

REPO = "pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit"

tokenizer = AutoTokenizer.from_pretrained(REPO)
# No load_in_4bit needed — the quantization_config is baked into config.json.
model = AutoModelForCausalLM.from_pretrained(REPO, device_map="auto")

messages = [
    {"role": "system", "content": "You are a precise assistant. Follow instructions exactly and return structured outputs when requested."},
    {"role": "user", "content": "Return a JSON object with keys: summary, risk_level, action_items."},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

out = model.generate(**{"input_ids": inputs}, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Tip: use add_generation_prompt=True. Without it, the ChatML template can leak a leading assistant/user role label into the decoded text.

Bare tool-call dispatch — enforce outer keys if your stack does not normalize

import json

SYS = (
    'Respond ONLY with a JSON object using EXACTLY these keys:\n'
    '{"tool_name": "add", "args": {"a": <integer>, "b": <integer>}}\n'
    'No other text, no markdown.'
)

def call_tool(prompt: str, retries: int = 3) -> dict:
    for attempt in range(retries):
        msgs = [{"role": "system", "content": SYS}, {"role": "user", "content": prompt}]
        ids = tokenizer.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
        out = model.generate(**{"input_ids": ids}, max_new_tokens=64, do_sample=False)
        raw = tokenizer.decode(out[0][ids.shape[-1]:], skip_special_tokens=True).strip()
        try:
            parsed = json.loads(raw)
            assert "tool_name" in parsed and "args" in parsed
            assert "a" in parsed["args"] and "b" in parsed["args"]
            return parsed
        except (json.JSONDecodeError, AssertionError, KeyError):
            if attempt == retries - 1:
                raise ValueError(f"Tool call failed after {retries} attempts. Raw: {raw}")

print(call_tool("Add 5 and 10."))

Serving — vLLM (OpenAI-compatible)

vllm serve pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit \
  --quantization bitsandbytes \
  --max-model-len 8192

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-required")
resp = client.chat.completions.create(
    model="pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit",
    messages=[{"role": "user", "content": "Your prompt here"}],
    temperature=0.3,
)
print(resp.choices[0].message.content)

Recommended Generation Settings

temperature: 0.3 · top_p: 0.9 for general use; temperature: 0.0 (greedy) for deterministic structured/tool output.

---

Evaluation Artifacts

The full per-case evaluation CSV (comparison_results_v7_21_Qwen2.5_32B_Instruct_4bit_20260527_233818_20260629_002808.csv) and rollup.json are published in this repository for independent verification.

---

Artifact Provenance

Quantized from Qwen/Qwen2.5-32B-Instruct using BitsAndBytes (NF4, double-quant, float16 compute). The FP16 base was not evaluated due to VRAM constraints.

---

Evaluation Methodology

quant_eval v7.21 — proprietary behavioral evaluation harness, PBH Applied Systems.

Fixture set: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c079371b02a94f3c149eb78a6adc03dc16ff6833b964fbf4174f0)

Evaluation hardware: NVIDIA RTX 4090 (24 GB) · Runner: quantized_bitsandbytes (4-bit only) · Evaluation date: June 29, 2026 · Seed: 42

---

About PBH Applied Systems

PBH Applied Systems, LLC is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development.

Patrick Hill — Founder & Principal AI/ML Systems Architect · Author of Applied Machine Learning: Concepts, Tools, and Case Studies (required reading, UAT CSC 373)

Core Service Areas: LLM Optimization & Deployment · AI Evaluation Frameworks · Agentic AI Infrastructure · Scalable AI Application Development · ML Pipeline Design & Analytics · Model & Agent Cataloging

---

🔬 About quant_eval & This Evaluation Series

quant_eval is a proprietary behavioral evaluation harness developed by PBH Applied Systems, LLC. It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under pbhappliedsystems has been independently evaluated using quant_eval before being recommended for any production role.

See it in action: Live AI Agent Demo →

Need a deployment recommendation? Not sure which quantization level — NF4, Q4_K_M, AWQ — is right for your hardware, latency target, or agent type? → pbhappliedsystems.com

---

Evaluated and published by PBH Applied Systems, LLC · patrick@pbhappliedsystems.com

---

📞 Work With PBH Applied Systems

This NF4 result is a concrete example of why backend-aware evaluation matters: the same 32B weights at 4-bit behave differently across BitsAndBytes and GGUF — cleaner tool-call output on one, stronger multi-step planning on the other. The evaluation tells you exactly which to deploy for which workload.

👉 Book a Scoping Call · 👉 Request an Evaluation Report — from $2,500

Connect

---

License

This repository inherits the license of the base model: Apache 2.0 — Qwen/Qwen2.5-32B-Instruct

The quant_eval evaluation methodology, fixture set, and scoring framework are proprietary to PBH Applied Systems, LLC and are not included in this repository.

---

BitsAndBytes quantization and behavioral evaluation performed by PBH Applied Systems, LLC · quant_eval v7.21 · Run ID: 20260629_002808