Sunflower-Speech

A speech-native large language model for Ugandan languages. Processes audio directly without cascaded ASR, enabling real-time spoken interaction with 55ms time-to-first-token. Includes GRPO post-training for improved transcription fidelity.

Architecture

Sunflower-Speech combines three components:

• Audio encoder: Whisper Large V3 fine-tuned on 10 Ugandan languages
• Multimodal projector: 2-layer MLP with 8× frame stacking (Ultravox architecture)
• Language model: Sunflower-32B (Qwen 3 32B adapted for Ugandan languages)

The projector maps Whisper encoder outputs directly into the LLM's embedding space, bypassing explicit transcription and eliminating cascade latency.

Supported Languages

Performance

Speech Translation (BLEU ↑)

Cascaded: Whisper transcription → Sunflower translation. Text-only: Sunflower translating ground-truth text (oracle upper bound).

Transcription (WER ↓, median %)

Whisper baseline uses a standard encoder-decoder trained on the same data. GRPO post-training reduces the model's tendency to paraphrase or respond conversationally.

Latency (A100-80GB)

Usage

import transformers   # version 4.51.3
import torch          # version 2.6.0+cu124
import numpy as np    # version 1.26.4
import librosa        # version 0.10.2.post1

pipe = transformers.pipeline(
    model="Sunbird/Sunflower-Speech",
    trust_remote_code=True,
    device="cuda",
    torch_dtype=torch.bfloat16,
    model_kwargs={"low_cpu_mem_usage": False},
)

path = "<path-to-input-audio>"  # TODO: pass the audio here
audio, sr = librosa.load(path, sr=16000)

turns = [
  {
    "role": "system",
    "content": "You are Sunflower, a helpful assistant made by Sunbird AI who understands all Ugandan languages. You specialise in accurate translations, explanations, summaries and other language tasks."
  },
  {
    "role": "user",
    "content": "Transcribe to English: <|audio|>",
  },
]

result = pipe(
    {"audio": audio, "sampling_rate": sr, "turns": turns},
    max_new_tokens=1024,
    do_sample=False,
    top_p=None,
    top_k=None
)

print(result)

Intended Use

• Real-time speech translation for Ugandan languages
• Voice-based instruction following and Q&A
• Accessible interfaces for primarily spoken languages

Limitations

• Transcription accuracy lower than standalone Whisper (trades accuracy for latency and flexibility)
• May paraphrase or respond conversationally instead of transcribing verbatim
• Requires A100-80GB for full-precision inference
• Evaluated on single-turn interactions only

Hardware Requirements

Citation

@inproceedings{akera2026realtime,
  title={Real-Time Spoken Instruction Following and Translation in Ugandan Languages},
  author={Akera, Benjamin and Hu, Tim Wenjie and Walukagga, Patrick and Ouma, Evelyn Nafula and Gilbert, Yiga and Mwebaze, Ernest Tonny and Quinn, John},
  booktitle={Proceedings of the 7th Workshop on African Natural Language Processing (AfricaNLP)},
  year={2026}
}

Acknowledgments

Developed by Sunbird AI. Built on the Ultravox architecture.