SHIVS 0.5B: voice command to phone action

SHIVS turns a short speech transcript, such as "switch the bluetooth on and open whatsapp", into a compact JSON array of Android phone actions. It is a LoRA fine-tune of Qwen2.5-0.5B-Instruct, shipped as a Q4_K_M GGUF (about 380 MB) and meant to run on the phone with a GBNF grammar that makes invalid output impossible.

It is the language half of an on-device voice assistant. The speech half is whisper.cpp (base.en is included in this repository for convenience).

Read the limitations first. The model itself is accurate on text. The measured weak point is speech-to-text on a phone microphone, which mishears names. End-to-end accuracy on live speech was 57 to 78% in tests with one speaker and one phone, and has not been measured for other voices, accents or phones. See Limitations.

What is in this repository

SHA-256: shivs-q4_k_m.gguf f71238c9353949f8d4ecdb79b04314ef59fa50d15160d7124f5856be4ed6f339; ggml-base.en.bin a03779c86df3323075f5e796cb2ce5029f00ec8869eee3fdfb897afe36c6d002.

What it does

Input: one transcript, already stripped of any wake word, for example Turn on the Bluetooth and open WhatsApp.

Output: a JSON array of 1 to 3 actions, no spaces, fixed key order:

[{"action":"bluetooth_toggle","parameters":{"state":"on"}},{"action":"open_app","parameters":{"app_name":"WhatsApp"}}]

System prompt (must match exactly):

You are SHIVS. Convert voice commands into a JSON array of phone actions. Output JSON only.

Chat format: ChatML (<|im_start|>system ... <|im_end|><|im_start|>user ... <|im_end|><|im_start|>assistant).

The 29 actions

clarify and unsupported are always the only action in the array. The schema is closed: anything else is rejected by the grammar.

The model only chooses the action and its parameters. The app decides how to carry each one out. Android does not let a normal app silently switch Wi-Fi, mobile data, Bluetooth, airplane mode, location, battery saver or the hotspot, so an app typically opens the right settings screen for those.

How to use it

With llama.cpp

llama-cli -m shivs-q4_k_m.gguf --grammar-file actions.gbnf --temp 0 -n 160 \
  -sys "You are SHIVS. Convert voice commands into a JSON array of phone actions. Output JSON only." \
  -p "switch the bluetooth on and open whatsapp"

With llama-cpp-python

from llama_cpp import Llama, LlamaGrammar

llm = Llama(model_path="shivs-q4_k_m.gguf", n_ctx=512, verbose=False)
grammar = LlamaGrammar.from_file("actions.gbnf")
SYSTEM = "You are SHIVS. Convert voice commands into a JSON array of phone actions. Output JSON only."

def parse(text):
    out = llm.create_chat_completion(
        messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": text}],
        temperature=0.0, max_tokens=160, grammar=grammar)
    return out["choices"][0]["message"]["content"]

print(parse("set an alarm for 6 15 pm and turn on wifi"))

Always use the grammar. Without it the model sometimes invents action names (set_volume_level, battery_toggle) or enum values. With it, the output is always valid JSON of the schema.

Speed tip: the system prompt and chat header are identical on every call. Decode them once and keep them in the KV cache; on the test phone this cut the model step from about 1.5 s to about 1.0 s.

Recommended pipeline on a phone

microphone, then whisper.cpp (base.en), then remove a leading wake word, then fix spoken digits (for example "double 5" to "55"), then this model with the grammar, then validate the JSON against the schema, then run the action.

A prompt that lists the user's contact and app names, given to whisper as its initial prompt, improved live accuracy noticeably in our tests (see below). For commands that contain digits, transcribing a second time without that prompt avoids letters appearing inside phone numbers.

Training

• Base model: Qwen/Qwen2.5-0.5B-Instruct (Apache-2.0).
• Method: LoRA (rank 32, alpha 32, all attention and MLP projections) with Unsloth on a Colab T4, learning rate 2e-4, 3 epochs. The adapter was merged and exported to GGUF, then quantised to Q4_K_M. The notebook may have overridden the script defaults; check shivs_train.ipynb in the project for the exact run.
• Data: about 16,000 training rows generated from templates (spoken phrasing variants, transcript-style noise such as dropped words and repeated words, wake-word variants, multi-action sentences, unsupported and ambiguous requests) plus about 4,700 paraphrases written by Claude Haiku 5.5 from template seeds. The language model never wrote labels: each paraphrase inherits its seed's label and is kept only if every name, number and free-text value from the seed appears verbatim.
• Two modes: a development mode that holds out every fifth template (to measure generalisation to unseen phrasing) and a ship mode that trains on all templates. This release is the ship model, so its test split is not independent of its templates; use the real-speech results below for generalisation.
• No real user data was used. Names in the data are drawn from name lists in the generator.

Evaluation

"Strict" means the whole JSON array must match exactly. "Name-tolerant" also accepts a contact or app name with the same first letter and at least 0.65 string similarity, which forgives speech-recognition spelling errors. Both are always shown; strict is the headline number.

Text input (the model alone)

JSON parse rate 100% and schema validity 100% with the grammar. Because this is the ship model, these numbers measure the model on phrasing it has seen the templates for.

Real speech through whisper.cpp (recorded read-aloud lines, wake word removed)

Q4_K_M with the grammar was within about two rows of the 16-bit model on every real-speech set (a difference at the level of noise for sets this size).

Live microphone on a phone (the real use case)

All with whisper.cpp base.en, this model with the grammar, on one phone, one speaker, one room. 45 fresh lines per set.

By type of line on set C: commands without a name about 92% right (24 of 26); commands with a contact name only 4 of 11, even with the name in the prompt; combined commands (two actions) 2 of 5.

What the errors are. The model copies whatever name whisper wrote, so the failure is the transcript ("Tarchan please" for "torch on please", "Carl Anita" for "call Anita"). Fed the text that was meant, the model got 12 of 13 two-action lines right. A larger whisper model (small.en) did not help on a fresh set.

Speed and cost on the test phone (CPU only): about 4.5 s per command in the foreground (speech-to-text 2.3 to 3.9 s, this model 1.7 to 2.1 s), about 1.1 GB of memory for the whole pipeline, about 1.4 mAh per command including the screen, 8 to 18 s to load the models at start.

Limitations and what has not been measured

• One speaker, one accent, one phone. Every live number above comes from a single person on a single vivo phone (Snapdragon 695, 5.5 GB). Accuracy for other voices, accents, genders, ages, noise levels and phones is unknown. Do not assume the figures carry over.
• English only. The training data and base.en are English. Indian English is the main accent tested; other accents are untested.
• Names are the weak point. Names whisper has never heard are often misspelled. Matching the heard name to the user's contacts with fuzzy and sound-alike matching helped by at most one line in 45 once the whisper prompt was on.
• Closed action set. The model only knows the 29 actions above. Requests outside them should produce unsupported, but were tested on a small set only.
• Numbers. Spoken digits are a known weak spot ("double 5", a dropped digit in a long phone number). The reference app repairs the model's number from the transcript's digits.
• Time. The model sometimes converts a 12-hour time wrongly (for example "eleven PM" became 11:00 in a test); check alarm times before relying on them.
• Combined commands (up to three actions) are handled well from clean text (12 of 13) but poorly after a noisy transcript.
• Latency. About 4.5 s per command on a mid-range phone is far above what a commercial assistant achieves. The original design target of about 0.4 s was not reachable with this stack on this phone.
• Background running. On the test phone the maker's power manager force-stopped the app when the screen was off and the phone was on battery, even with every standard exemption. Always-listening wake-word use was not achieved.
• No safety evaluation beyond the above. There has been no red-teaming, bias or fairness audit, or testing with children or speech impairments.

Intended use and misuse

Intended: converting short, clear voice commands into a fixed set of phone actions, inside an app that validates the JSON, asks before ambiguous or risky actions, and runs on the device.

Not intended, and not suitable without more work:

• Any setting where a wrong action causes harm (emergency calls, payments, medical or safety use).
• Acting on messages or calls to other people without a check. A misheard name can reach the wrong contact. A reference app guards this by asking for a spoken yes or no when a name matches a contact only weakly.
• General conversation. The model is trained to output JSON only and will refuse chit-chat with unsupported.

Privacy

This model runs entirely on the phone: speech, transcript and output stay on the device. The reference app needs microphone, contacts, messages and location permissions for the actions to work, and should explain this to users.

Licences and attribution

• Fine-tuned model: derived from Qwen2.5-0.5B-Instruct, Apache-2.0. Please keep the original licence and notices when redistributing.
• ggml-base.en.bin: converted OpenAI Whisper weights for whisper.cpp, MIT.
• Training paraphrases were generated with Claude Haiku 5.5; check the provider's current terms before building a commercial product that depends on that data.

Files, schema and reproducibility

The schema (actions.json), the grammar generator, the data generator and the evaluation scripts live in the SHIVS project repository, not here. The grammar in this repository was generated from that schema; the model, grammar and app must be updated together.

Contact

Maintainer: shivap1712 on Hugging Face (use the repository's Community tab for questions and reports).