Sunflower 32B - GGUF

GGUF quantized versions of the Sunflower model for Ugandan language translation tasks.

Model Details

• Base Model: Sunbird/Sunflower-32B
• Model Size: 32B parameters
• Architecture: Qwen2.5
• Quantization: K-means quantization with importance matrix
• Languages: English, Luganda, and other Ugandan languages

Available Files

Recommended Quantizations

Warning: Experimental Quantizations

The following quantizations achieve extreme compression but may significantly impact translation quality. Use for research and experimentation only.

Note: The experimental quantizations (IQ1_S, IQ2_XXS, TQ1_0) use advanced compression techniques that may not preserve the specialized knowledge for Ugandan language translation. Test thoroughly before production use.

Additional Files

Usage

llama.cpp

# Download model
huggingface-cli download Sunbird/Sunflower-32B-GGUF sunflower-32B-q4_k_m.gguf --local-dir .

# Run inference
./llama-cli -m sunflower-32B-q4_k_m.gguf -p "Translate to Luganda: Hello, how are you today?"

Ollama Integration

Ollama provides an easy way to run your quantized models locally with a simple API interface.

Installation and Setup

# Install Ollama (Linux/macOS)
curl -fsSL https://ollama.ai/install.sh | sh

# Or download from https://ollama.ai for Windows

# Start Ollama service (runs in background)
ollama serve

Creating Modelfiles for Different Quantizations

Q4_K_M (Recommended) - Modelfile:

cat > Modelfile.q4 << 'EOF'
FROM ./gguf_outputs/model-q4_k_m.gguf

# System prompt for your specific use case
SYSTEM """You are a linguist and translator specializing in Ugandan languages, made by Sunbird AI."""

# Chat template (adjust for your base model architecture)
TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
{{ .Response }}<|im_end|>"""

# Stop tokens
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"

# Generation parameters
PARAMETER temperature 0.3
PARAMETER top_p 0.95
PARAMETER top_k 40
PARAMETER repeat_penalty 1.1
PARAMETER num_ctx 4096
PARAMETER num_predict 500
EOF

Experimental IQ1_S - Modelfile:

cat > Modelfile.iq1s << 'EOF'
FROM ./gguf_outputs/model-iq1_s.gguf

SYSTEM """You are a translator for Ugandan languages. Note: This is an experimental ultra-compressed model - quality may be limited."""

# Same template and parameters as above
TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
{{ .Response }}<|im_end|>"""

PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.3
PARAMETER top_p 0.95
PARAMETER num_ctx 2048  # Smaller context for experimental model
EOF

Importing Models to Ollama

# Import Q4_K_M model (recommended)
ollama create sunflower-32B:q4 -f Modelfile.q4

# Import experimental IQ1_S model
ollama create sunflower-32B:iq1s -f Modelfile.iq1s

# Import other quantizations
ollama create sunflower-32B:q5 -f Modelfile.q5
ollama create sunflower-32B:q6 -f Modelfile.q6

# Verify models are imported
ollama list

Expected output:

NAME                    ID              SIZE    MODIFIED
sunflower-32B:q4        abc123def       17GB   2 minutes ago
sunflower-32B:iq1s      def456ghi       6.2GB   1 minute ago

Using Ollama Models

Interactive Chat:

# Start interactive session with Q4 model
ollama run sunflower-32B:q4

# Example conversation:
# >>> Translate to Luganda: Hello, how are you today?
# >>> Give a dictionary definition of the Samia term "ovulwaye" in English
# >>> /bye (to exit)

# Start with experimental model
ollama run sunflower-32B:iq1s

Single Prompt Inference:

# Quick translation with Q4 model
ollama run sunflower-32B:q4 "Translate to Luganda: People in villages rarely accept new technologies."

# Test experimental model
ollama run sunflower-32B:iq1s "Translate to Luganda: Good morning"

# Dictionary definition
ollama run sunflower-32B:q4 'Give a dictionary definition of the Samia term "ovulwaye" in English'

Ollama API Usage

Start API Server:

# Ollama automatically serves API on http://localhost:11434
# Test API endpoint
curl http://localhost:11434/api/version

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(model_path="sunflower-32B-q4_k_m.gguf")
result = llm("Translate to Luganda: How are you?")
print(result['choices'][0]['text'])

Performance Notes

• Q4_K_M: Recommended for most use cases
• Q5_K_M: Better quality with moderate size increase
• Q6_K: High quality for production use
• Q8_0: Near-lossless quality

Technical Details

Quantized using llama.cpp with importance matrix calibration for optimal quality preservation.

License

Apache 2.0