An optimized new version of ISSAI's audio-vision-language model Qolda-AVL-5B.
Developed by Almubdi Mutaikhan with Batyr Arystanbekov (ISSAI), under the supervision of Prof. Huseyin Atakan Varol (ISSAI). See Citation.
ISSAI earlier released issai/Qolda-AVL-5B: a 5B audio-vision-language model for Kazakh, Russian and English. It extends Qwen3-VL-4B-Thinking with an audio branch: a fine-tuned Whisper encoder and an audio projection. All three modalities are adapted to Kazakh in staged training.
This model is its next version. It keeps Qolda-AVL's audio encoder and audio training data, and changes three things:
Result: stronger text and vision than Qolda-AVL-5B, better Russian and English speech recognition and better speech translation. On audio question answering (SAKURA, SpokenMQA) it is 2.7 points behind Qolda-AVL-5B. The numbers are in Comparison. How it was built is in Implementation.
The name "Qolda" reflects both the design and the purpose of the model family in Kazakh: "in hand" (қолда) for its compact accessibility, and "to support" (қолдау) for its assistive nature.
Contents: Model details · Implementation · Comparison · How to use · Limitations · Citation · Contributions
| type | chat model with thinking (<think> … </think>); input: text, image, audio; output: text |
| language model + vision | issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer, unchanged (4.58 B) |
| audio encoder | Whisper-large-v3-turbo encoder, ISSAI's Kazakh fine-tune from Qolda-AVL-5B, frozen (0.64 B) |
| audio connector | trained, 13 M |
| audio-only LoRA | r = 16, α = 32, 30.5 M; on only for prompts with audio |
| parameters | 5.26 B in total |
| audio input | one clip per prompt, 0.3–30 s, any sample rate (resampled to 16 kHz) |
| precision | bfloat16 |
| languages | Kazakh, Russian, English |
Qolda-AVL-5B's Whisper-large-v3-turbo encoder (ISSAI's Kazakh fine-tune), copied unchanged and kept frozen. It gives 50 frames per second of audio.
average-pool ×4 → LayerNorm → Linear → GELU → Linear: 4 Whisper frames become one audio token, so one second of
audio is 12.5 tokens in the language model. A second input reads Whisper's layer 18 through its own LayerNorm and a
projection that starts at zero. Middle layers of Whisper keep more speaker information than its last layer: a
linear probe tells speaker gender on SAKURA with 93% accuracy from layer 18, but only 51% from the last layer.
A LoRA adapter (r = 16) on the attention, linear-attention and MLP projections of all 32 layers of the language model. It is switched on for prompts with audio and off otherwise. With it off, the model is the chat model itself: text and image logits are bit-identical (maximum difference 0.0 on Kazakh, Russian, English and image prompts).
The language model's own weights and the audio encoder stay frozen throughout.
| stage | trained | data | settings |
|---|---|---|---|
| 1 | connector | one pass over 2.22 M examples | learning rate 1e-4, cosine; 50% of answers with thinking |
| 2 | connector + layer-18 input + LoRA | 425 k examples | learning rate 5e-5 (connector), 1e-4 (LoRA) |
ISSAI's Qolda-AVL audio training set, plus our own self-distilled spoken instructions. One pass:
| group | examples | share |
|---|---|---|
| speech recognition, English / Russian / Kazakh (Kazakh ×2) | 652 k / 558 k / 132 k | 29% / 25% / 12% |
| language identification (17 languages) | 244 k | 11% |
| speech translation, English → Kazakh | 134 k | 6% |
| spoken instructions (answers written by the chat model from the transcript) | 134 k | 6% |
| everyday sounds, music | 63 k (×2), 44 k | 6%, 2% |
| gender, emotion, animal sounds (×3) | 30 k | 4% |
1% of the rows were held out for testing. Training clips that also occur in the SAKURA test set were removed.
Three models: Qwen3.5-4B (no audio), ISSAI's earlier Qolda-AVL-5B, and this model.
| Qwen3.5-4B | Qolda-AVL-5B | this model | |
|---|---|---|---|
| language model | Qwen3.5-4B | Qwen3-VL-4B-Thinking, trained further | issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer, frozen |
| tokens per word, Kazakh / Russian / English | 3.48 / 1.89 / 1.25 | — | 1.76 / 1.72 / 1.25 |
| audio tokens per second | — | 50 | 12.5 |
| Kazakh text average (7 benchmarks, below) | 76.44 | 70.88 | 78.33 |
| vision average (9 benchmarks, below) | 74.2 | 68.2 | 74.2 |
| audio question answering (20 tasks, below) | — | 68.3 | 65.6 |
SAKURA (Kazakh and Russian, single- and multi-hop) and SpokenMQA (Kazakh spoken maths), from ISSAI's audio benchmark collection. All items (SAKURA 497–500 per task, SpokenMQA 2,168), our runs of both models with the same prompts and settings: thinking on, temperature 0.7, top-p 0.95, top-k 20, up to 2,048 new tokens (4,096 for SpokenMQA), answer in JSON. An answer that is missing or cannot be parsed counts as wrong. Accuracy in %.
| category | tasks | Qolda-AVL-5B | this model |
|---|---|---|---|
| language | 4 | 90.5 | 90.0 |
| emotion | 4 | 37.0 | 37.5 |
| gender | 4 | 73.8 | 72.7 |
| animal | 4 | 53.8 | 48.1 |
| spoken maths (SpokenMQA) | 4 | 86.3 | 79.7 |
| average of 20 tasks | 68.3 | 65.6 |
| task | language | hops | Qolda-AVL-5B | this model |
|---|---|---|---|---|
| SAKURA · Language | Kazakh | single | 95.2 | 94.0 |
| Kazakh | multi | 85.4 | 84.4 | |
| Russian | single | 95.2 | 95.6 | |
| Russian | multi | 86.0 | 86.0 | |
| SAKURA · Emotion | Kazakh | single | 41.4 | 38.2 |
| Kazakh | multi | 32.2 | 34.2 | |
| Russian | single | 38.4 | 41.8 | |
| Russian | multi | 35.8 | 35.8 | |
| SAKURA · Gender | Kazakh | single | 74.8 | 75.0 |
| Kazakh | multi | 71.8 | 71.0 | |
| Russian | single | 74.4 | 75.6 | |
| Russian | multi | 74.0 | 69.2 | |
| SAKURA · Animal | Kazakh | single | 57.7 | 49.7 |
| Kazakh | multi | 49.1 | 45.5 | |
| Russian | single | 57.5 | 46.5 | |
| Russian | multi | 51.1 | 50.7 | |
| SpokenMQA · short digit | Kazakh | 92.0 | 81.0 | |
| SpokenMQA · long digit | Kazakh | 86.0 | 77.9 | |
| SpokenMQA · single-step reasoning | Kazakh | 90.9 | 86.3 | |
| SpokenMQA · multi-step reasoning | Kazakh | 76.3 | 73.7 |
Note on the sound tasks: the Qolda-AVL audio training set contains 459 of the 500 SAKURA AnimalQA clips and 282 of the 500 EmotionQA clips (from public sound and emotion datasets). We removed them from our training, so for this model they are unseen.
The 1% hold-out of the audio training data (500 clips per language; 300 for translation), never used in our training. Same prompt for both models. This model: thinking off (the recommended mode for transcription). Qolda-AVL-5B: its default thinking mode. Scored on the clips where Qolda-AVL-5B gave an answer within 2,048 new tokens (4,096 for translation): all 500 Kazakh, 494 English, 467 Russian and 263 translation clips.
| task | Qolda-AVL-5B | this model |
|---|---|---|
| Kazakh speech recognition, WER ↓ | 10.8% | 10.9% |
| Russian speech recognition, WER ↓ | 11.3% | 4.7% |
| English speech recognition, WER ↓ | 11.6% | 6.1% |
| English speech → Kazakh text, chrF ↑ | 40.3 | 48.0 |
Rule-scored benchmarks, our runs of all three models with the same harness and settings (thinking on, temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5). Qolda-AVL-5B was run as its vision-language part. Accuracy in %.
| benchmark | lang | n | Qwen3.5-4B | Qolda-AVL-5B | this model |
|---|---|---|---|---|---|
| MMBench | kk | 4,329 | 87.3 | 87.3 | 88.5 |
| OCRBench | kk | 441 | 31.5 | 23.6 | 32.0 |
| AI2D | kk | 3,088 | 78.2 | 72.8 | 79.9 |
| MMStar | kk | 1,499 | 71.4 | 64.4 | 70.7 |
| MMStar | ru | 1,500 | 62.4 | 56.1 | 62.2 |
| MMBench | en | 4,377 | 89.7 | 88.3 | 89.6 |
| OCRBench | en | 1,000 | 82.3 | 74.5 | 82.0 |
| AI2D | en | 3,088 | 87.5 | 79.0 | 87.4 |
| MMStar | en | 1,500 | 77.1 | 67.5 | 75.2 |
These settings are Qwen3.5's recommended ones. Qolda-AVL-5B's own card reports its results with its own settings, and some of them are higher than in our run (for example MMStar-kk 67.5, OCRBench-kk 30.6).
Thinking on, full test sets, Kazakh benchmarks from ISSAI's Qolda language suite. Qwen3.5-4B and this model: our runs (temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5). Qolda-AVL-5B: from its model card, which does not state its settings, so this part is for reference. Accuracy in %.
| benchmark | lang | Qwen3.5-4B | Qolda-AVL-5B | this model |
|---|---|---|---|---|
| KazMMLU | kk | 72.5 | 69.3 | 78.4 |
| MMLU | kk | 77.7 | 73.1 | 79.2 |
| MMLU-Pro | kk | 70.0 | 62.2 | 70.0 |
| MMLU-Redux | kk | 80.6 | 76.2 | 81.1 |
| GPQA | kk | 61.7 | 47.9 | 62.1 |
| GSM8K | kk | 88.0 | 85.8 | 90.3 |
| Belebele | kk | 84.6 | 81.7 | 87.2 |
| Kazakh average | 76.44 | 70.88 | 78.33 | |
| MMLU | en | 85.9 | 79.1 | 84.5 |
| MMLU-Pro | en | 79.4 | 71.3 | 77.6 |
| GSM8K | en | 96.2 | 95.2 | 96.1 |
All text results of this model, with more benchmarks, are on the card of issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer. They hold here unchanged, because text answers are identical.
Requirements: transformers 5.x (tested with 5.12.1), torch, peft, soundfile, scipy, pillow.
One GPU with about 13 GB of memory.
pip install torch "transformers>=5" peft soundfile scipy pillow
Download the model and load it with the provided loader:
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("issai/Qolda-AVL-5B-OptimizedTokenizer")
sys.path.insert(0, path)
from load_avl import load, ask, ask_audio
model = load()
The model thinks by default (<think> … </think> before the answer); pass think=False to answer directly.
print(ask(model, "y = (lnx)^2 функциясының туындысын тап. JSON форматында жауап бер: {'answer': '...'}"))
print(ask(model, "Суретті егжей-тегжейлі сипаттап бер. Неше жылқы көріп тұрсың және олардың түстері қандай?",
image=f"{path}/assets/sample_image.jpg"))
Give a text instruction together with the audio, describing what to do with it.
prompt = """Математикалық есепті шеш.
Respond ONLY with this JSON format: {"explanation": "<your step-by-step reasoning>", "answer": <integer or float number>}
The answer must be a number (integer or float). No text, no units, just the number.
"""
print(ask_audio(model, f"{path}/assets/sample_audio.wav", prompt))
# transcription: thinking off is faster
print(ask_audio(model, f"{path}/assets/sample_audio.wav", "Осы аудиодағы сөйлеуді мәтінге айналдырыңыз.", think=False))
From the shell: python load_avl.py clip.wav "Transcribe this audio."
Serving text and images with vLLM: the folder is also a normal Qwen3.5 checkpoint (the audio files are ignored),
so vllm serve issai/Qolda-AVL-5B-OptimizedTokenizer --reasoning-parser qwen3 serves text and image prompts. Audio
needs load_avl.py.
| file | what |
|---|---|
model.safetensors, config.json, tokenizer*, chat_template.jinja, *preprocessor_config.json |
the chat model with vision, unchanged |
audio_encoder/ |
Whisper encoder weights and feature-extractor settings |
audio_connector.safetensors |
the connector |
audio_lora.safetensors |
the audio-only LoRA |
avl_config.json, load_avl.py, avl_code/ |
how the parts fit together, and the loader |
assets/ |
the architecture figure and the sample image and audio used above |
load_avl.py); vLLM serves only text and images.Apache-2.0. Built on Qwen/Qwen3.5-4B and the audio encoder of issai/Qolda-AVL-5B. The training data have their own licenses.
@misc{mutaikhan2026qoldaavl5bopt,
title = {Qolda-AVL-5B-OptimizedTokenizer},
author = {Mutaikhan, Almubdi and Arystanbekov, Batyr and Varol, Huseyin Atakan},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/issai/Qolda-AVL-5B-OptimizedTokenizer}}
}
The audio encoder and audio training data come from Qolda-AVL. Please also cite:
@article{qolda-avl-bdcc,
AUTHOR = {Arystanbekov, Batyr and Maxutov, Akylbek and Nurimanov, Aspandiyar and Varol, Huseyin Atakan},
TITLE = {Extending a Vision–Language Model with Audio Understanding: Introducing Qolda-AVL for the Kazakh Language},
JOURNAL = {Big Data and Cognitive Computing},
VOLUME = {10},
YEAR = {2026},
NUMBER = {6},
ARTICLE-NUMBER = {192},
URL = {https://www.mdpi.com/2504-2289/10/6/192},
ISSN = {2504-2289},
DOI = {10.3390/bdcc10060192}
}
model.safetensors 9.2 GBaudio_encoder/encoder.safetensors 1.3 GBaudio_lora.safetensors 122 MBaudio_connector.safetensors 52 MBissai/Qolda-AVL-5B