ShaziLLM-1B ONNX INT8

Low-memory ONNX release of ShaziLLM-1B. The decoder includes KV-cache inputs and outputs for efficient autoregressive generation and follows the Hugging Face onnx/ repository layout.

• Parameters: 1.08B
• Format: ONNX opset 18
• Quantization: dynamic per-channel signed INT8
• Sensitive output head: retained in FP32 to reduce quantization drift
• File size: approximately 1.6 GB
• Context configuration: up to 131,072 tokens; use much shorter contexts on constrained hardware

Transformers.js

npm install @huggingface/transformers

import { pipeline } from "@huggingface/transformers";

const generator = await pipeline(
  "text-generation",
  "Sharjeelbaig/ShaziLLM-1B-ONNX",
  { dtype: "int8" },
);

const messages = [
  { role: "user", content: "Explain recursion in one short paragraph." },
];
const prompt = generator.tokenizer.apply_chat_template(messages, {
  tokenize: false,
  add_generation_prompt: true,
  enable_thinking: false,
});

const output = await generator(prompt, {
  max_new_tokens: 128,
  do_sample: false,
  return_full_text: false,
});
console.log(output[0].generated_text);

Node.js is the most predictable low-memory JavaScript target. Browser use requires enough memory for the 1.6 GB graph plus runtime buffers; use the WASM backend and limit context length.

Python with Optimum ONNX Runtime

pip install "optimum-onnx[onnxruntime]" transformers

from transformers import AutoTokenizer, pipeline
from optimum.onnxruntime import ORTModelForCausalLM

model_id = "Sharjeelbaig/ShaziLLM-1B-ONNX"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = ORTModelForCausalLM.from_pretrained(
    model_id,
    subfolder="onnx",
    file_name="model_int8.onnx",
    use_cache=True,
)
generator = pipeline("text-generation", model=model, tokenizer=tokenizer)

messages = [{"role": "user", "content": "What is 7 times 8?"}]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
result = generator(
    prompt,
    max_new_tokens=64,
    do_sample=False,
    return_full_text=False,
)
print(result[0]["generated_text"])

Runtime compatibility note

The model has an explicit attention head_dim of 128 even though its internal residual hidden size is 1536. Some current ONNX generation wrappers initialize cache tensors using hidden_size / num_attention_heads and ignore the explicit head_dim. This ONNX-only repository reports a runtime projection width of 2048 so those wrappers allocate the correct 128-wide cache. The exported graph and normal BF16 checkpoint retain the actual 1536 residual width. This compatibility value must not be used to reconstruct or train the architecture.

Validation

• Cached FP32 export completed with maximum observed logit difference of approximately 0.00119 against PyTorch.
• INT8 generation was tested through Python Optimum and Transformers.js pipelines.
• Identity, arithmetic, and factual probes were exercised; runtime-specific wording can differ.
• INT8 generation is not bit-identical to BF16. Use ShaziLLM-1B when output fidelity matters more than memory.

Limitations

Quantization can change token choices, especially when logits are close. The configured 128K context is not realistic on low-memory systems because KV-cache memory grows with sequence length. Start with short contexts and conservative generation limits. Validate generated code and never grant model-produced tool calls authority without deterministic checks.

License and attribution

Released under Apache License 2.0; see LICENSE. ShaziLLM-1B is a fine-tuned derivative of OpenBMB/MiniCPM5-1B. The ShaziLLM name identifies this release and does not imply authorship of the upstream base architecture.