Sunbird/Sunflower-32B - 4-bit Quantized (FP4)

This is a 4-bit quantized version of Sunbird/Sunflower-32B using FP4 (Float Point 4) quantization with BitsAndBytes.

Model Description

🌻 Sunflower-32B is a multilingual language model developed by Sunbird AI for Ugandan languages. Built on the Qwen 3-32B architecture, the model supports translation and text generation across 31 Ugandan languages plus English. The model achieves the highest translation accuracy among evaluated models in 24 of 31 language pairs.

Developed by: Sunbird AI Model type: Causal language model Languages: 31 Ugandan languages + English (see language codes above)

This quantized version reduces memory requirements by approximately 75% while maintaining most of the original model's performance, making it suitable for:

• Consumer-grade GPUs (24-32GB VRAM)
• Local inference deployments
• Cost-effective cloud deployments
• Rapid prototyping and development

Quantization Details

• Quantization Method: FP4 (Float Point 4)
• Bits: 4-bit
• Double Quantization: Enabled
• Compute dtype: bfloat16
• Library: BitsAndBytes

What is FP4?

FP4 (Float Point 4) uses a standard 4-bit floating-point representation. While it provides good compression, NF4 typically offers better quality for language models.

Advantages of FP4:

• Standard floating-point representation
• Good for weights with diverse distributions
• Slightly faster on some hardware

Usage

Installation

pip install torch transformers accelerate bitsandbytes

Usage

Installation

pip install torch transformers accelerate bitsandbytes

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("Sunbird/Sunflower-32B-4bit-fp4-bnb")

# Configure quantization (already quantized, but need config for loading)
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="fp4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "Sunbird/Sunflower-32B-4bit-fp4-bnb",
    quantization_config=quantization_config,
    device_map="auto",
    trust_remote_code=True
)

Chat Format

SYSTEM_MESSAGE = """You are Sunflower, a multilingual assistant for Ugandan languages made by Sunbird AI.
You specialise in accurate translations, explanations, summaries and other cross-lingual tasks."""
prompt_text = "Sunbird AI is a non-profit research organization in Kampala, Uganda. We build and deploy practical applied machine learning systems for African contexts."
user_prompt = f"Translate to Luganda, {prompt_text}"

messages = [
    {"role": "system", "content": SYSTEM_MESSAGE},
    {"role": "user", "content": user_prompt}
]


prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([prompt], return_tensors="pt").to(model.device)
text_streamer = transformers.TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

print(f'\n\n{user_prompt}\n---')

# For better quality outputs
num_beams = 5
generation_config = {
    "max_new_tokens": 512,
    "temperature": 0.3,
    "do_sample": True,
    "no_repeat_ngram_size": 5,
    "num_beams": num_beams,
}


outputs = model.generate(
  **inputs,
  **generation_config,
  streamer = text_streamer if num_beams==1 else None,
)

if num_beams > 1:
    response = tokenizer.decode(outputs[0][len(inputs['input_ids'][0]):], skip_special_tokens=True)
    print(response)

Output

Translate to Luganda, Sunbird AI is a non-profit research organization in Kampala, Uganda. We build and deploy practical applied machine learning systems for African contexts.
---

Sunbird AI kitongole kya kunoonyereza ekitali kya magoba ekisangibwa mu Kampala, Uganda ekizimba n'okuteeka mu nkola enkola za kompyuta ezikozesebwa mu mbeera za Afirika.

Performance Considerations

Memory Requirements

Note: Actual memory usage may vary based on sequence length, batch size, and context window.

Speed vs Quality Trade-offs

• Inference Speed: Slightly slower than full precision due to dequantization overhead (~10-20%)
• Quality: Minimal degradation for most tasks (~2-5% performance drop on benchmarks)
• Best For: Inference on consumer hardware, development, and testing

System Requirements

Minimum Requirements

• GPU: NVIDIA GPU with compute capability 7.0+ (e.g., V100, RTX 2080, T4)
• VRAM: 24 GB
• CUDA: 11.0 or higher
• RAM: 32 GB

Recommended Requirements

• GPU: NVIDIA RTX 3090/4090, A100, or newer
• VRAM: 32+ GB
• CUDA: 12.0 or higher
• RAM: 48 GB

Limitations

• Quantization may introduce small numerical errors
• Not suitable for training (inference only)
• May show slightly degraded performance on tasks requiring high numerical precision
• Requires CUDA-compatible GPU (CPU inference not optimized)

Comparison with Other Quantization Methods

License

This model inherits the license from the base model: Sunbird/Sunflower-32B

Citation

If you use this model, please cite both the base model and BitsAndBytes:

@software{bitsandbytes,
  title = {BitsAndBytes},
  author = {Dettmers, Tim},
  url = {https://github.com/TimDettmers/bitsandbytes},
  year = {2023}
}

Acknowledgments

• Base model: Sunbird/Sunflower-32B
• Quantization: BitsAndBytes
• Framework: Hugging Face Transformers

Contact

For issues specific to this quantized version, please open an issue on the model repository. For issues with the base model, please refer to the original model repository.

Known Issues

None reported yet.

Changelog

• v1.0: Initial release