Sunbird/Sunflower-32B - 8-bit Quantized

This is an 8-bit quantized version of Sunbird/Sunflower-32B using BitsAndBytes quantization.

Model Description

🌻 Sunflower-32B is a multilingual language model developed by Sunbird AI for Ugandan languages. Built on the Qwen 3-32B architecture, the model supports translation and text generation across 31 Ugandan languages plus English. The model achieves the highest translation accuracy among evaluated models in 24 of 31 language pairs.

Developed by: Sunbird AI Model type: Causal language model Languages: 31 Ugandan languages + English (see language codes above)

This quantized version reduces memory requirements by approximately 50% while maintaining near-identical performance to the original model, making it ideal for:

• Mid-range GPUs (24GB-48GB VRAM)
• Production deployments with memory constraints
• Balanced performance and efficiency
• Large-scale inference services

Quantization Details

• Quantization Method: 8-bit Integer Quantization
• Bits: 8-bit
• Library: BitsAndBytes
• Performance: Near-zero degradation from the original model

What is 8-bit Quantization?

8-bit quantization converts the model's 16-bit or 32-bit floating-point weights to 8-bit integers. This reduces memory usage by approximately 50% while maintaining excellent quality through careful calibration and outlier detection techniques implemented in BitsAndBytes.

Advantages of 8-bit:

• Minimal quality loss (typically <1% on benchmarks)
• Faster than 4-bit quantization
• Better numerical stability
• Suitable for production environments

Usage

Installation

pip install torch transformers accelerate bitsandbytes

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("Sunbird/Sunflower-32B-8bit-bnb")

# Configure quantization (already quantized, but need config for loading)
quantization_config = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_threshold=6.0,  # Threshold for outlier detection
)

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "Sunbird/Sunflower-32B-8bit-bnb",
    quantization_config=quantization_config,
    device_map="auto",
    trust_remote_code=True
)

Chat Format

SYSTEM_MESSAGE = """You are Sunflower, a multilingual assistant for Ugandan languages made by Sunbird AI.
You specialise in accurate translations, explanations, summaries and other cross-lingual tasks."""
prompt_text = "Sunbird AI is a non-profit research organization in Kampala, Uganda. We build and deploy practical applied machine learning systems for African contexts."
user_prompt = f"Translate to Luganda, {prompt_text}"

messages = [
    {"role": "system", "content": SYSTEM_MESSAGE},
    {"role": "user", "content": user_prompt}
]


prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([prompt], return_tensors="pt").to(model.device)
text_streamer = transformers.TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

print(f'\n\n{user_prompt}\n---')

# For better quality outputs
num_beams = 5
generation_config = {
    "max_new_tokens": 512,
    "temperature": 0.3,
    "do_sample": True,
    "no_repeat_ngram_size": 5,
    "num_beams": num_beams,
}


outputs = model.generate(
  **inputs,
  **generation_config,
  streamer = text_streamer if num_beams==1 else None,
)

if num_beams > 1:
    response = tokenizer.decode(outputs[0][len(inputs['input_ids'][0]):], skip_special_tokens=True)
    print(response)

Output

Translate to Luganda, Sunbird AI is a non-profit research organization in Kampala, Uganda. We build and deploy practical applied machine learning systems for African contexts.
---

Sunbird AI kitongole kya kunoonyereza ekitali kya magoba ekisangibwa mu Kampala, Uganda ekizimba n'okuteeka mu nkola enkola za kompyuta ezikozesebwa mu mbeera za Afirika.

Performance Considerations

Memory Requirements

Note: Actual memory usage may vary based on sequence length, batch size, and context window.

Speed vs Quality Trade-offs

• Inference Speed: Comparable to full precision (~5-10% slower)
• Quality: Near-identical to original (<1% degradation)
• Best For: Production deployments requiring high quality with memory constraints

System Requirements

Minimum Requirements

• GPU: NVIDIA GPU with compute capability 7.0+ (e.g., V100, RTX 2080, T4)
• VRAM: 16 GB
• CUDA: 11.0 or higher
• RAM: 32 GB

Recommended Requirements

• GPU: NVIDIA RTX 3090/4090, A100, or newer
• VRAM: 24+ GB
• CUDA: 12.0 or higher
• RAM: 64 GB

Limitations

• Quantization may introduce minimal numerical errors
• Not suitable for training (inference only)
• Requires CUDA-compatible GPU (CPU inference not optimized)
• Slightly slower than full precision due to dequantization overhead

Comparison with Other Quantization Methods

When to Use 8-bit vs 4-bit

Choose 8-bit if:

• You need the best quality possible while saving memory
• You're deploying in production
• You have 16+ GB VRAM
• Speed is important

Choose 4-bit if:

• You're limited to 8-12 GB VRAM
• You're doing development/testing
• Slight quality reduction is acceptable
• Maximum memory savings are needed

License

This model inherits the license from the base model: Sunbird/Sunflower-32B

Citation

If you use this model, please cite both the base model and BitsAndBytes:

@software{bitsandbytes,
  title = {BitsAndBytes},
  author = {Dettmers, Tim},
  url = {https://github.com/TimDettmers/bitsandbytes},
  year = {2023}
}

Acknowledgments

• Base model: Sunbird/Sunflower-32B
• Quantization: BitsAndBytes
• Framework: Hugging Face Transformers

Contact

For issues specific to this quantized version, please open an issue on the model repository. For issues with the base model, please refer to the original model repository.