Huihui-Qwen-AgentWorld-35B-A3B-abliterated UD-Q3_K_M GGUF
An importance-matrix UD-Q3_K_M GGUF quant of
huihui-ai/Huihui-Qwen-AgentWorld-35B-A3B-abliterated.
Why this quant exists: the BF16 source is roughly 65-67 GB and is not practical on 22 GB cards. This UD-Q3_K_M build is 15.5 GiB / 16.65 GB, leaving enough working room for KV cache and compute buffers on a single 22 GB RTX 2080 Ti while preserving the abliterated AgentWorld model behavior for local agent experiments.
| File | Size | BPW |
|---|---|---|
Huihui-Qwen-AgentWorld-35B-A3B-abliterated-UD-Q3_K_M.gguf |
15.5 GiB / 16.65 GB | 3.84 |
Lineage
- Base: Qwen/Qwen-AgentWorld-35B-A3B — Qwen AgentWorld 35B-A3B MoE model for agent and environment-simulation use cases.
- Abliteration: huihui-ai/Huihui-Qwen-AgentWorld-35B-A3B-abliterated, described upstream as an uncensored/abliterated variant with reduced refusal behavior.
- This quant:
UD-Q3_K_M, produced with llama.cpp from the local F16 GGUF conversion of Huihui's BF16 release.
Quick Stats
| Metric | Value |
|---|---|
| Source GGUF | Huihui-Qwen-AgentWorld-35B-A3B-abliterated-F16.gguf |
| Quantizer | llama.cpp Vulkan build b9870 |
| Quant type | Q3_K_M with imatrix + tensor map |
| GGUF ftype | Q3_K - Medium |
| Output file size | 16,660,085,760 bytes |
| Output file size | 15.52 GiB / 16.66 GB |
| Reported BPW | 3.84 |
| Imatrix entries | 510 |
| Imatrix chunks | 76 |
Tensor type counts:
| Type | Count |
|---|---|
F32 |
361 |
IQ3_XXS |
78 |
IQ4_XS |
39 |
Q6_K |
15 |
Q8_0 |
240 |
This tensor map follows the public UD-style Q3_K_M layout used for Qwen-AgentWorld. A few additional attention output tensors stayed at Q6_K, making this build slightly more conservative than the smallest possible Q3 variant.
llama.cpp Server
Recommended launch:
llama-server \
-m Huihui-Qwen-AgentWorld-35B-A3B-abliterated-UD-Q3_K_M.gguf \
-ngl 99 \
-c 8192 \
--parallel 1 \
--jinja \
--reasoning on \
--reasoning-format deepseek \
--temp 0.6 \
--top-p 0.95 \
--top-k 20
For a pinned single-GPU Vulkan setup, expose only the intended GPU before launch, for example:
CUDA_VISIBLE_DEVICES= GGML_VK_VISIBLE_DEVICES=1 llama-server \
-m Huihui-Qwen-AgentWorld-35B-A3B-abliterated-UD-Q3_K_M.gguf \
--host 0.0.0.0 \
--port 8083 \
-ngl 99 \
-c 8192 \
--parallel 1 \
--jinja \
--reasoning on \
--reasoning-format deepseek
Important runtime note: run the server with reasoning parsing enabled. In local llama.cpp testing, server-level --reasoning off could still lead this model to emit Thinking Process: as ordinary message.content. The more reliable route was:
- Start llama.cpp with
--reasoning on --reasoning-format deepseek. - For direct final-answer calls, pass request-level
chat_template_kwargs: {"enable_thinking": false}. - For thinking-mode calls, omit that override and read reasoning from
message.reasoning_content.
Example direct call body:
{
"model": "huihui-qwen-agentworld-35b-a3b-abliterated-ud-q3km",
"messages": [
{"role": "user", "content": "Write five concise deployment checks."}
],
"chat_template_kwargs": {"enable_thinking": false},
"temperature": 0.2,
"max_tokens": 256
}
Local Smoke Test
Test host: dual RTX 2080 Ti, llama.cpp Vulkan backend, model loaded on one 22 GB card.
| Test | Result |
|---|---|
| Load on one RTX 2080 Ti | OK |
VRAM after load at -c 8192 |
~15.8 GiB |
| N=3 direct decode speed | median 82.32 tok/s, range 82.26-82.55 tok/s |
| N=3 prompt eval speed | median 319.84 tok/s |
| Thinking-mode probe | OK; reasoning separated into message.reasoning_content |
| JSON extraction | OK; valid JSON and array fields preserved |
| Agent patch-plan prompt | OK; exact five-bullet output |
| Tool routing JSON | OK; four valid routes |
| Chinese agent triage | OK; six concise checks |
| Abliterated/directness sanity | OK; 3/3 safe direct prompts had no refusal phrases |
These are smoke tests, not a standardized benchmark. Quality should be validated against your own agent workload before relying on the quant in production.
Quantization Recipe
The imatrix was generated from a small synthetic calibration set focused on terminal, repo-editing, JSON extraction, tool-call style transcripts, Chinese assistant turns, and safe fictional directness prompts.
llama-imatrix \
-m Huihui-Qwen-AgentWorld-35B-A3B-abliterated-F16.gguf \
-f huihui-agentworld-calibration.txt \
-o imatrix-huihui-agentworld-76.gguf \
--output-format gguf \
--chunks 76 \
-c 512 \
-ngl auto \
--split-mode none \
--main-gpu 0 \
--no-ppl
llama-quantize \
--imatrix imatrix-huihui-agentworld-76.gguf \
--tensor-type-file ud-q3km-tensor-types.txt \
Huihui-Qwen-AgentWorld-35B-A3B-abliterated-F16.gguf \
Huihui-Qwen-AgentWorld-35B-A3B-abliterated-UD-Q3_K_M.gguf \
Q3_K_M 24
The following helper files are included for reproducibility:
| File | Purpose |
|---|---|
quantization/imatrix-huihui-agentworld-76.gguf |
Importance matrix used for this quant |
quantization/ud-q3km-tensor-types.txt |
Tensor type map |
quantization/huihui-agentworld-calibration.txt |
Synthetic calibration text |
Abliteration Sanity
The upstream model is an abliterated build. Local safe smoke tests checked for functional directness using non-graphic fictional and agentic prompts. With request-level chat_template_kwargs: {"enable_thinking": false}, the model returned direct content without refusal phrases in all three abliterated sanity prompts.
This is not a safety guarantee. It is only a practical sanity check that the quantized GGUF still behaves like the abliterated upstream in ordinary local testing.
License
Inherits from the upstream model: Apache-2.0.
⚠ Safety: The upstream Huihui model card warns that refusal behavior and safety filtering are significantly reduced. Use this model in controlled environments and review outputs carefully.
Credits
- Source model: Qwen Team / Alibaba
- Abliteration: huihui-ai
- Quantization: this checkpoint, by coolthor
- Tooling: llama.cpp
📝 Quantized & benchmarked by ai-muninn — writeups on how it was built and how it actually runs.
Made by Thor Lin. Full benchmarks, setup notes and the runs that failed are on ai-muninn.com. More quantized models: huggingface.co/coolthor.
Provenance
This section documents release 3b61b0a3ae596fe60d6477f65e6e6f6a05f39eae, inspected on 2026-11-14. Where the exact upstream commit used at build time is unknown, it is listed as not recorded rather than backfilled with today's upstream HEAD.
Source lineage
| Source repository | Revision used for this build | Role |
|---|---|---|
| huihui-ai/Huihui-Qwen-AgentWorld-35B-A3B-abliterated | not recorded | Immediate BF16 input, converted locally to F16 GGUF |
Build details
Tools and versions: llama.cpp Vulkan build b9870; exact converter/quantizer git commit not recorded.
The model was quantized to Q3_K_M using an importance matrix and per-tensor overrides. The repository includes the synthetic calibration text, a 76-chunk/510-entry imatrix and a tensor-type map. The local note reports 16,649,095,680 output bytes whereas the card reports 16,660,085,760; the hash inventory below follows the captured HF release, rather than either earlier size claim.
The existing records cover quantization and format conversion only; they don't mention any fine-tuning or merging by coolthor. That reflects the documentation, not an independent weight diff.
Build script (copied from the existing card, not re-run): Because it references the repo by name without a pinned commit, the exact upstream commit used at build time is unknown.
llama-imatrix \
-m Huihui-Qwen-AgentWorld-35B-A3B-abliterated-F16.gguf \
-f huihui-agentworld-calibration.txt \
-o imatrix-huihui-agentworld-76.gguf \
--output-format gguf \
--chunks 76 \
-c 512 \
-ngl auto \
--split-mode none \
--main-gpu 0 \
--no-ppl
llama-quantize \
--imatrix imatrix-huihui-agentworld-76.gguf \
--tensor-type-file ud-q3km-tensor-types.txt \
Huihui-Qwen-AgentWorld-35B-A3B-abliterated-F16.gguf \
Huihui-Qwen-AgentWorld-35B-A3B-abliterated-UD-Q3_K_M.gguf \
Q3_K_M 24
Weight file SHA-256 checksums
The values below are Hugging Face LFS object SHA-256 values for the pinned release above. File sizes and digests agree with pinned tree metadata. They identify full file contents, not LFS pointer text or Git blob IDs. Calibration imatrices are included when present.
| File | Bytes | SHA-256 |
|---|---|---|
Huihui-Qwen-AgentWorld-35B-A3B-abliterated-UD-Q3_K_M.gguf |
16660085760 | 79eab4afc9cd79555ead3ee2ec66cae8efdd90e1df7b445cf1741105d04ec2f1 |
quantization/imatrix-huihui-agentworld-76.gguf |
192223904 | 7bfba4a6dbb888e1156045aa856a877ae7ae50fa4be13918b7ad9dff26493d8e |
Verifying the files
Download the desired files from revision 3b61b0a3ae596fe60d6477f65e6e6f6a05f39eae and compare sha256sum <file> (Linux) or shasum -a 256 <file> (macOS) with the table. If cloning with Git LFS, hash the downloaded weight files, not the LFS pointer files. A matching hash confirms your files match this release; it does not prove the weights came only from the listed base model with no other changes.
Checking how the weights were derived requires the exact upstream commit, tool versions, recipe and any calibration or training data, then re-running the build and diffing the weights with a format-aware tool (or comparing file hashes where serialization is deterministic). Because of the gaps listed below, this build can't be reproduced exactly today. Quantized files are expected to differ numerically from BF16 sources, and a raw file hash cannot establish a base-to-quant weight diff. No quantization rerun or weight-diff audit was performed for this provenance review.
Known gaps: Base revision; Exact tool commit and initial BF16 → F16 conversion invocation; Historical output-size discrepancy.
Sources: existing model card and HF metadata.