PsarAI-Omni v0.1
Speech in, speech out, with one model thinking in between, for Khmer, English and French. There is no ASR, translation or text-to-speech model in the loop: Gemma-4-E2B-it hears the question through its own audio encoder, writes a short spoken reply, and a speech decoder turns Gemma's hidden states straight into speech units for a fast HiFi-GAN vocoder (Jarvis voice, or any voice from a few seconds of reference audio).
question audio ─► Gemma-4-E2B-it audio encoder + LoRA adapter (Khmer / English / French speech)
─► Gemma writes the reply (voice-assistant system prompt), sentence by sentence
each sentence ─► base Gemma final hidden states + token embeddings ─► speech decoder ─► mHuBERT units
─► durations + pitch ─► unit HiFi-GAN ─► 24 kHz speech (Jarvis or cloned voice)
GGUF version (llama.cpp, one server, voice cloning): nphearum/PsarAI-Omni-GGUF.
Listen
A spoken question goes in; Omni hears it, writes a reply and speaks it, sentence by sentence. Questions are from the FLEURS test set (CC-BY-4.0). Cloned is the same reply in a voice cloned from one 9 s reference clip (young man, deep and smooth)
| Question | Reply (Jarvis) | Reply (cloned voice) | |
|---|---|---|---|
| Khmer | មនុស្សមួយចំនួនគិតថា គាត់និយាយត្រូវ ប៉ុន្តែមនុស្សជាច្រើនជឿថាគាត់និយាយមិនត្រូវទេ។ គាត់ថាប្រព័ន្ធព្រះអាទិត្យ ធ្វើដំណើរវិលជុំវិញផែនដី រួមទាំងព្រះអាទិត្យ (និងផ្កាយដទៃទៀតផងដែរ)។ |
នោះជាទស្សនៈមួយដែលគួរឱ្យចាប់អារម្មណ៍។ គំនិតនេះបង្ហាញពីការយល់ដឹងផ្សេងៗគ្នាអំពីធម្មជាតិ។ |
នោះជាទស្សនៈមួយដែលគួរឱ្យចាប់អារម្មណ៍។ គំនិតនេះបង្ហាញពីការយល់ដឹងផ្សេងៗគ្នាអំពីធម្មជាតិ។ |
| English | All nouns, alongside the word Sie for you, always begin with a capital letter, even in the middle of a sentence. |
That is a common style in some writing. It is a stylistic choice rather than a strict rule. |
That is a common style in some writing. It is a stylistic choice rather than a strict rule. |
| French | L’accident a eu lieu en terrain montagneux, et il semblerait que cela ait été causé par un incendie malveillant. |
Ceci est une information importante. L'accident s'est produit en montagne et un incendie est suspecté. |
Ceci est une information importante. L'accident s'est produit en montagne et un incendie est suspecté. |
Usage
python psarai_omni.py question.wav --out reply.wav # --voice ref.wav to clone, --gemma PATH for a local Gemma
from psarai_omni import Omni
omni = Omni() # downloads google/gemma-4-E2B-it unless Omni(gemma=PATH)
text, wav = omni.chat("question.wav") # float32 at omni.sample_rate (24000)
for sentence, wav in omni.stream("question.wav", question="What is the speaker talking about? Answer in English."):
play(wav) # first sentence plays while Gemma is still writing
voice = omni.voice("reference.wav")
wav = omni.say("សួស្តី លោក។ Good evening, sir.", voice=voice)
Needs torch, transformers with Gemma 4, peft, torchvision (Gemma 4 processor), safetensors, numpy,
soundfile, soxr; for voice cloning also torchaudio, pyworld, librosa, scipy, einops, diffusers. About 10 GB of GPU memory at peak with the per-layer embedding table kept in
CPU RAM (the default).
Results (v0.1)
| value | |
|---|---|
| Khmer speech transcription by the adapted Gemma (FLEURS, CER) | 12.8% (base Gemma 30.7%) |
| Khmer speech → English translation (FLEURS, chrF) | 34.9 (base Gemma 24.9) |
| English / French transcription (FLEURS, CER) | 4.0% / 5.0% |
| spoken reply heard back by an ASR vs Gemma's reply text, Khmer (CER, n=4) | 7.3% |
| spoken reply heard back by an ASR vs Gemma's reply text, English (CER, n=10) | 11.7% |
| spoken reply heard back by an ASR vs Gemma's reply text, French (CER, n=2) | 7.7% |
| first audio after the question ends (median, RTX 3090) | 0.63 s |
The speech decoder alone, on held-out sentences: Jarvis-style 4.2% CER, Gemma-style replies 12.0% (PsarAI-Speech T2U reading the same text: 4.0% / 10.1%).
Files
| File | What |
|---|---|
adapter/ |
LoRA on Gemma's audio encoder (r32) and LM attention (r8), audio projections trained in full |
speech_decoder.safetensors |
Gemma hidden states + token embeddings ─► mHuBERT units (CTC), durations, pitch (66M) |
vocoder.safetensors, voices.safetensors |
speaker-conditioned unit HiFi-GAN and the Jarvis embedding (from PsarAI-Speech) |
campplus.safetensors, campplus.py |
CAM++ speaker encoder for voice cloning |
flow_clone.safetensors, mhubert/, s3gen/ |
clone v3 from PsarAI-Speech: a cloned voice's reply is rendered by a flow-matching decoder that continues the reference recording (sounds like the person, ~0.4 s more per sentence); Jarvis stays on the fast HiFi-GAN |
psarai_omni.py, psarai_speech.py |
inference code (psarai_speech.py provides the T2U and vocoder classes) |
Training
- Audio adapter: stage 1, transcription of Khmer (DDD, FLEURS, Jarvis, designed voices), English and French speech (6000 steps); stage 2, 45% transcription + 55% instructions (reply, describe, summarize, translate) whose targets were written by text-only Gemma from the transcript (self-distillation, 3000 steps).
- Speech decoder: initialized from the PsarAI-Speech T2U unit decoder; trained on 34k Jarvis clips (Gemma teacher-forced on their text) plus 31k of Gemma's own reply sentences with units from the PsarAI-Speech T2U.
Limitations
- Research release. Gemma's Khmer replies are weak: they often restate what was said instead of answering.
- Audio input is limited to 30 s (Gemma 4). Replies are short by design (system prompt).
- Base model terms: Gemma Terms of Use apply to the adapter and to outputs of the combined model.
License
Gemma Terms of Use for the adapter and the combined model. The speech parts come from PsarAI-Speech
(CC-BY-NC-4.0, non-commercial); mhubert/ is utter-project/mHuBERT-147
(CC-BY-NC-SA-4.0); s3gen/ code and the S3Gen / HiFT / CAM++ weights in flow_clone.safetensors /
campplus.safetensors come from ResembleAI/chatterbox (MIT).