Ornith 1.5 9B FP8 Heretic

Ornith-1.5-9B-FP8-Heretic is a standalone, merged, compressed-tensors FP8 derivative of Sohailhosseini/Ornith-1.5-9B-FP8, modified with Heretic.

This repository contains the complete model. It is not a LoRA adapter: no separate PEFT adapter and no local copy of the base model are required at inference time.

What changed

The release pipeline was:

1. Run a bounded Heretic optimization against the pinned FP8 base checkpoint.
2. Export the selected rank-1 LoRA direction.
3. Merge the adapter into a BF16 intermediate.
4. Re-quantize the merged model to the source checkpoint's compressed-tensors FP8 scheme.
5. Verify the standalone export tensor-by-tensor and reload it without PEFT or the base-model directory.

Abliteration is intended to reduce learned refusal behavior. It does not make the model universally correct, safe, unbiased, or suitable for every deployment.

Base checkpoint and provenance

The adapter-only and native merged runs selected identical parameters and scores.

Quantization

The derivative preserves the source checkpoint's compressed-tensors quantization boundary and metadata:

• Format: compressed-tensors float-quantized
• Quantization method: FP8
• Weights: FP8 E4M3, static per-channel
• Input activations: FP8, dynamic per-token
• Quantization target: Linear
• KV-cache quantization: none
• Protected/ignored module boundary: identical to the pinned source checkpoint

Serialized tensor inventory

model.safetensors is 11,910,387,016 bytes, the same serialized tensor-file size as the pinned source FP8 model.

Bounded derivative verification

A tensor-by-tensor comparison against the pinned source checkpoint found:

Every changed model weight is within a Heretic target:

• mlp.down_proj
• self_attn.o_proj
• linear_attn.out_proj

Protected BF16 islands and unrelated quantized tensors remain byte-identical to the source. This bounds the release as a narrow derivative of the pinned FP8 checkpoint rather than an uncontrolled whole-model conversion.

Runtime validation

The final repository tree was loaded directly with AutoModelForImageTextToText.from_pretrained, without PEFT and without access to the base-model directory.

Verified properties:

• Model class: Qwen3_5ForConditionalGeneration
• Quantization status: compressed
• Remaining LoRA parameters: 0
• Deterministic CUDA generation completed successfully
• Prompt: The capital of France is
• Generated continuation: Paris

Coherence and tool-calling smoke test

A small deterministic, greedy-decoding smoke suite passed 6/6 cases:

Observed native tool-call outputs included:

• get_weather(city="Chicago", unit="fahrenheit")
• calculator(expression="17 * 23")
• Dependency-aware first step: geocode(place="Eiffel Tower") before requesting weather

After receiving a simulated weather response, the model incorporated its temperature, condition, and observation time into a natural-language answer without issuing another tool call.

This is a smoke test, not a comprehensive agent benchmark. Multi-step real-tool execution, parallel calls, malformed tool results, tool restraint, long-context behavior, multimodal quality, factuality, and broad safety behavior remain unevaluated.

Usage

Use recent releases of Transformers, Accelerate, and compressed-tensors with Qwen3.5 support.

Load the model

from transformers import AutoModelForImageTextToText, AutoProcessor

repo_id = "cbert33/Ornith-1.5-9B-FP8-Heretic"

model = AutoModelForImageTextToText.from_pretrained(
    repo_id,
    dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(repo_id)

Text generation

from transformers import AutoTokenizer

repo_id = "cbert33/Ornith-1.5-9B-FP8-Heretic"
tokenizer = AutoTokenizer.from_pretrained(repo_id)

messages = [
    {"role": "user", "content": "Explain why the sky appears blue in two sentences."}
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=False,
)
print(tokenizer.decode(outputs[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Native tool-call prompting

The included chat template accepts OpenAI-style function definitions through its tools argument and emits Qwen's native XML tool-call representation.

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"},
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["city", "unit"],
            },
        },
    }
]

messages = [
    {
        "role": "user",
        "content": "What is the current weather in Chicago in Fahrenheit?",
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)

Your inference framework is responsible for parsing the generated tool call, executing the function, and returning the tool result to the model.

Tested environment

• Windows 11 with WSL2
• NVIDIA GeForce RTX 5080 (16 GB)
• CUDA 13.0
• PyTorch 2.13.0+cu130
• Transformers 5.6.2
• compressed-tensors 0.18.0
• llmcompressor 0.13.0

Memory note

In the tested direct Transformers path, compressed-tensors materialized/decompressed weights during loading and peak CUDA allocation reached approximately 19.0 GB. The model completed inference on the 16 GB RTX 5080 through WSL unified memory, but generation was slow. An FP8-aware serving runtime should be qualified separately before production deployment.

Limitations

• The Heretic search was deliberately bounded: two positive prompts, two negative prompts, two KL prompts, and one optimization trial.
• The model has not received a comprehensive text, vision, benchmark, safety, or refusal evaluation after modification.
• The small coherence/tool test demonstrates only the explicitly tested cases.
• Abliteration may increase compliance with harmful, incorrect, or poorly specified requests.
• Quantization and runtime support vary by GPU architecture and inference backend.
• Users are responsible for evaluating legality, safety, reliability, and fitness for their intended use.

Integrity

The repository includes SHA256SUMS. The model tensor hash is:

8650b2f6cf35afb133c7225c65b25483211ccbf72901fe9cdd7803c64c32c0af  model.safetensors

License

The source checkpoint is published under the MIT license. This derivative is distributed under the same license. Users must also comply with the licenses and terms applicable to upstream model components and dependencies.