MDL-0.6B

Related work: 🛠️ MiDashengLM · 📚 ACAVCaps Dataset · 📊 MECAT Benchmark

This repository contains the GGUF versions of mispeech/midashenglm-0.6b-fp32. Can be used with llama.cpp for efficient local inference.

Usage

[!IMPORTANT] This model currently requires our llama.cpp fork to run. See the build instructions to get started.

Try it in your browser

We have a WebAssembly demo that runs this model entirely on-device using your CPU — no server required.

One-shot inference

llama-cli --model backbone-bf16.gguf --mmproj mmproj-f32.gguf \
  --reasoning off --temp 0 --audio audio.wav \
  --prompt 'Write a detailed caption.' \
  --single-turn --display-prompt

Interactive mode

llama-cli --model backbone-bf16.gguf --mmproj mmproj-f32.gguf \
  --reasoning off --temp 0

For more CLI options, see the CLI documentation.

OpenAI/Anthropic-compatible API server

llama-server --model backbone-bf16.gguf --mmproj mmproj-f32.gguf

For more server options, see the server documentation.

Results

The following tables present the preliminary evaluation results of MDL-0.6B. We compare our compact model against the baseline MiDashengLM-7B-1021, as well as two multimodal large language models: Qwen2.5-Omni-7B and Kimi-Audio-Instruct.

Audio Captioning Results

We first evaluate on MECAT-Caption, which organizes captions into three strands. Systemic Captions comprise a concise short caption centered on the primary audio content and a long caption that adds contextual detail and how events interact. Content-Specific Captions use three branches—speech, music, and sound events—evaluated independently; the table reports pure vs. mixed variants for each. The Content-Unrelated Caption strand focuses on acoustic properties (e.g., recording quality and reverberation) rather than semantic scene content. Metrics are reported with DATE↑. Beyond MECAT-Caption, we report standard music captioning on MusiCaps and SongDescriber (FENSE↑) and general environmental / audio captioning on AudioCaps (dev), Clotho (test), and AutoACD (test) (FENSE↑).

If you are interested in the datasets and evaluation used for audio caption training and benchmarking, see ACAVCaps and MECAT.

Metrics: Higher is better. "-" denotes data not available.

Audio and Paralinguistic Classification

Metrics: Higher is better.

ASR Performance

Metrics: WER/CER (lower is better).

Question Answering Results

We first evaluate MECAT-QA, which targets reasoning and assessment over audio (subsets such as direct perception, sound characteristics, quality assessment, environment reasoning, inference / judgment, and application-oriented content), reported with DATE. Notably, the compact MDL-0.6B model improves markedly on these MECAT-QA subsets compared to the 7B MiDashengLM-7B-1021 model, reflecting strong zero-shot reasoning, logical inference, and fine-grained analysis despite its smaller size.

MMAU-Pro follows, using the Answer task and ACC↑ across capability splits (IF, Multi-Audio, Music, Open-Ended, Sound, cross-modal combinations, Spatial, Speech, Voice, and the overall Average).

Finally, we include additional QA benchmarks: AudioCaps-QA and MusicQA (FENSE↑), and MuChoMusic (ACC↑).

Metrics: Higher is better. "-" denotes data not available.

Citation

If you find MDL-0.6B useful in your research or business applications, please cite the underlying work it builds on: MiDashengLM (efficient audio understanding with general audio captions) and DashengTokenizer (unified continuous audio tokenization for understanding and generation).

@techreport{midashenglm7b,
  title      = {MiDashengLM: Efficient Audio Understanding with General Audio Captions},
  author     = {{Horizon Team, MiLM Plus}},
  institution= {Xiaomi Inc.},
  year       = {2025},
  note       = {Contributors: Heinrich Dinkel et al. (listed alphabetically in Appendix B)},
  url        = {https://arxiv.org/abs/2508.03983},
  eprint     = {2508.03983},
}

@misc{dinkel2026dashengtokenizer,
  title        = {DashengTokenizer: One layer is enough for unified audio understanding and generation},
  author       = {Heinrich Dinkel and Xingwei Sun and Gang Li and Jiahao Mei and Yadong Niu and Jizhong Liu and Xiyang Li and Yifan Liao and Jiahao Zhou and Junbo Zhang and Jian Luan},
  year         = {2026},
  eprint       = {2602.23765},
  archivePrefix= {arXiv},
  primaryClass = {cs.SD},
  url          = {https://arxiv.org/abs/2602.23765},
}

If you are interested in caption datasets and evaluation, see ACAVCaps and MECAT.

@misc{niu2026acavcaps,
  title         = {ACAVCaps: Enabling large-scale training for fine-grained and diverse audio understanding},
  author        = {Yadong Niu and Tianzi Wang and Heinrich Dinkel and Xingwei Sun and Jiahao Zhou and Gang Li and Jizhong Liu and Junbo Zhang and Jian Luan},
  year          = {2026},
  eprint        = {2603.24038},
  archivePrefix = {arXiv},
  primaryClass  = {eess.AS},
  url           = {https://arxiv.org/abs/2603.24038},
}

@misc{niu2025mecat,
  title         = {MECAT: a multi-experts constructed benchmark for fine-grained audio understanding tasks},
  author        = {Yadong Niu and Tianzi Wang and Heinrich Dinkel and Xingwei Sun and Jiahao Zhou and Gang Li and Jizhong Liu and Xiyang Liu and Junbo Zhang and Jian Luan},
  year          = {2025},
  eprint        = {2507.23511},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2507.23511},
}