Astrea-R8-Chat-9B-Heretic-T32-GGUF

GGUF conversions of Altworld/Astrea-R8-Chat-9B, modified using the Heretic automated directional-ablation process.

This release is based on Heretic and includes several quantization levels for use with llama.cpp, LM Studio, KoboldCpp, and other GGUF-compatible runtimes.

The model retains its original multimodal architecture. Vision use requires the separate matching mmproj file included in this repository.

Model details

Practical context length depends on available memory, runtime configuration, and workload. A smaller context such as 8,192 or 16,384 tokens is a reasonable starting point for local use.

What was changed?

The source model was processed using p-e-w/heretic, which applies optimized directional ablation—sometimes called abliteration—to reduce learned refusal behavior.

Heretic searches for ablation parameters that reduce refusals while minimizing divergence from the original model. This is not a conventional supervised fine-tune, LoRA, merge, or prompt modification.

This release uses the output selected as KL divergence: 0.0316, Refusals: 8/100.

The language-model weights were modified. The vision encoder and multimodal projector were not intentionally ablated.

Important MTP note

The source configuration declared an additional Multi-Token Prediction or NextN layer, but the saved checkpoint did not contain the associated MTP tensors.

Including that declaration caused affected llama.cpp builds to expect a nonexistent tensor:

blk.32.attn_norm.weight

The language-model GGUFs in this repository were therefore exported with:

--no-mtp

This excludes the unavailable MTP head. It does not remove normal text generation, chat, reasoning, or vision capabilities.

Available files

Actual filenames may differ slightly depending on the final files uploaded to this repository.

Quantization recommendations

Best overall compact version

Q4_K_M language model
Q8_0 mmproj

Better text quality

Q5_K_M or Q6_K language model
Q8_0 mmproj

Maximum practical quality

Q8_0 language model
Q8_0 or BF16 mmproj

Archival/reference version

BF16 language model
BF16 mmproj

The mmproj is much smaller than the language model and directly affects visual understanding. Q8_0 or BF16 is recommended for the projector rather than aggressively quantizing it.

Text-only usage with llama.cpp

llama-cli \
  -m Astrea-R8-Chat-9B-Heretic-T32-Q4_K_M.gguf \
  -ngl 999 \
  -c 8192 \
  -n 512 \
  -cnv

For a one-shot prompt:

llama-cli \
  -m Astrea-R8-Chat-9B-Heretic-T32-Q4_K_M.gguf \
  -ngl 999 \
  -c 8192 \
  -n 512 \
  -p "Explain why the sky appears blue."

On Windows CMD, place the command on one line or use ^ instead of \ for line continuation.

Vision usage with llama.cpp

Vision requires both the language-model GGUF and the matching mmproj:

llama-cli \
  -m Astrea-R8-Chat-9B-Heretic-T32-Q4_K_M.gguf \
  --mmproj mmproj-Astrea-R8-Chat-9B-Heretic-T32-Q8_0.gguf \
  --image example.jpg \
  -ngl 999 \
  -c 8192 \
  -n 512 \
  -p "Describe this image in detail."

You may also use the dedicated multimodal test utility:

llama-mtmd-cli \
  -m Astrea-R8-Chat-9B-Heretic-T32-Q4_K_M.gguf \
  --mmproj mmproj-Astrea-R8-Chat-9B-Heretic-T32-Q8_0.gguf \
  --image example.jpg \
  -ngl 999 \
  -c 8192 \
  -p "Describe this image."

Server usage

Text and vision can be exposed through llama.cpp’s OpenAI-compatible server:

llama-server \
  -m Astrea-R8-Chat-9B-Heretic-T32-Q4_K_M.gguf \
  --mmproj mmproj-Astrea-R8-Chat-9B-Heretic-T32-Q8_0.gguf \
  -ngl 999 \
  -c 8192 \
  --host 0.0.0.0 \
  --port 8080

Omit --mmproj when only text generation is needed.

Binding to 0.0.0.0 exposes the server to the local network. Use appropriate firewall rules and authentication controls before exposing it beyond a trusted network.

Chat template

The model’s embedded chat template was preserved during conversion.

GGUF-compatible applications should normally detect and use the embedded template automatically. Avoid manually selecting an unrelated Qwen template unless your runtime fails to detect it.

Evaluation

This release has not yet been subjected to a comprehensive independent benchmark suite.

The selected Heretic candidate was chosen from the automated optimization process, but that should not be treated as evidence of performance on general reasoning, coding, multilingual, multimodal, or safety benchmarks.

Suggested evaluations include:

• Comparison against the unmodified source model
• KL-divergence or perplexity comparison
• Instruction-following tests
• Refusal-rate testing
• General reasoning and coding tasks
• Vision-question answering
• OCR and detailed image-description tests
• Long-context retrieval
• Quantization comparisons across Q4_K_S through Q8_0

Community benchmark results and reproducible test reports are welcome.

Intended use

This model is intended for:

• Local experimentation
• Research into directional ablation and refusal behavior
• General conversational use
• Creative writing
• Coding and technical assistance
• Multimodal image understanding
• Comparing quantization formats
• Evaluating alignment and behavioral changes

Limitations

Directional ablation can change behavior in ways that are not fully captured by refusal-rate or divergence measurements.

Possible limitations include:

• Reduced safety refusals
• Harmful, offensive, misleading, or inappropriate output
• Incorrect factual claims
• Overconfidence
• Degraded behavior on prompts unlike those used during optimization
• Differences between quantization levels
• Reduced visual accuracy when using a quantized projector
• Inconsistent tool-call or structured-output behavior
• High memory use at very long context lengths

The absence of a refusal does not mean a response is correct, safe, legal, or appropriate.

Users are responsible for evaluating the model for their own application and implementing any safeguards required for that application.

Safety notice

This is a deliberately less-restricted model.

It may comply with requests that the upstream model would decline, and it may generate content that is inaccurate, offensive, unsafe, or unsuitable for deployment without additional controls.

Do not rely on the model as the sole source of medical, legal, financial, security, or other high-stakes advice.

Reproducibility

The language model was converted from the selected Heretic output using llama.cpp.

Representative BF16 conversion:

python convert_hf_to_gguf.py PATH_TO_HERETIC_OUTPUT \
  --no-mtp \
  --outfile Astrea-R8-Chat-9B-Heretic-T32-BF16-noMTP.gguf \
  --outtype bf16

Representative direct Q8_0 conversion:

python convert_hf_to_gguf.py PATH_TO_HERETIC_OUTPUT \
  --no-mtp \
  --outfile Astrea-R8-Chat-9B-Heretic-T32-Q8_0.gguf \
  --outtype q8_0

Representative Q4_K_M quantization from BF16:

llama-quantize \
  Astrea-R8-Chat-9B-Heretic-T32-BF16-noMTP.gguf \
  Astrea-R8-Chat-9B-Heretic-T32-Q4_K_M.gguf \
  Q4_K_M

Representative Q8_0 mmproj conversion:

python convert_hf_to_gguf.py PATH_TO_HERETIC_OUTPUT \
  --mmproj \
  --outfile Astrea-R8-Chat-9B-Heretic-T32-Q8_0.gguf \
  --outtype q8_0

The converter automatically adds the mmproj- filename prefix when exporting a multimodal projector.

All lower-precision language-model quants should be generated independently from the BF16 master rather than from another quantized file.

Credits

• Original model: Altworld/Astrea-R8-Chat-9B
• Underlying architecture: Qwen3.5
• Directional-ablation tooling: p-e-w/heretic
• GGUF conversion and runtime: ggml-org/llama.cpp

This repository distributes a modified and quantized derivative. It does not claim authorship of the original model, architecture, training data, or the Heretic and llama.cpp tooling.

License

This repository is licensed under the Apache License 2.0, following the license of the upstream model.

This is a modified derivative of:

• Altworld/Astrea-R8-Chat-9B
• Qwen/Qwen3.5-9B

The modifications, GGUF conversion, and quantization performed for this repository do not imply endorsement by the original model authors.

See the included LICENSE file for the complete license terms.

Disclaimer

This model is provided without warranties. The uploader is not responsible for outputs produced by the model or for downstream uses by third parties.