MOSS-Music

English | 简体中文

MOSS-Music is an open-source music understanding model from

MOSI.AI, the OpenMOSS team,

and Shanghai Innovation Institute. Built on the same

audio backbone as MOSS-Audio,

MOSS-Music is further specialised on music via dedicated continual pre-training

and supervised fine-tuning — targeting **musical captioning, lyrics ASR,

structural analysis, chord / key / tempo reasoning, and long-form musical

question answering. In this release, we provide two 8B models**:

MOSS-Music-8B-Instruct and MOSS-Music-8B-Thinking. The Instruct variant

is optimised for direct instruction following on musical prompts, while the

Thinking variant provides stronger chain-of-thought reasoning for musical

analysis.

News

• 2026.05.01: 🎉🎉🎉 We have released MOSS-Music.
• 2026.05.01: 🎉🎉🎉 We have released MOSS-Music-Data-Pipeline for large-scale music data annotation and processing.

Contents

• Introduction
• Model Architecture
• DeepStack Cross-Layer Feature Injection
• Time-Aware Representation
• Released Models
• Music Data Pipeline
• Evaluation
• Quickstart
• Environment Setup
• SGLang Serving
• Gradio App
• More Information
• LICENSE
• Citation

Introduction

Music is not just audio plus lyrics — understanding it requires perceiving

harmonic structure, rhythm, timbre, instrumentation, performance nuance, and

the textual content of the lyrics, and reasoning about them jointly across

time. MOSS-Music is built to unify these capabilities within a single

model.

• Lyrics ASR & time-aligned transcription: Accurate singing ASR with

sentence- and word-level timestamps, robust to backing tracks.

• Musical captioning & tagging: Natural-language descriptions of mood,

genre, instrumentation, production style, and emotional trajectory.

• Key / tempo / chord reasoning: Identifies musical key, beats, downbeats,

and chord progressions, including timestamped chord transcription.

• Structural analysis: Segments a song into intro / verse / chorus /

bridge / outro and reasons about repetition and contrast.

• Instrument & voice recognition: Identifies prominent instruments and

singing voices (solo / chorus, gender, register).

• Musical QA and long-form analysis: Open-ended question answering

grounded in a full track, including chain-of-thought reasoning in the

Thinking variant.

Model Architecture

MOSS-Music inherits the MOSS-Audio modular design, comprising three

components: an audio encoder, a modality adapter, and a large language model.

Raw audio is first encoded by MOSS-Audio-Encoder into continuous temporal

representations at 12.5 Hz, which are then projected into the language

model's embedding space through the adapter and finally consumed by the LLM

for auto-regressive text generation.

Rather than relying on off-the-shelf audio frontends, we train a dedicated

encoder from scratch to obtain more robust acoustic representations, tighter

temporal alignment, and better extensibility across musical styles, singing,

and non-speech acoustic content.

DeepStack Cross-Layer Feature Injection

Using only the encoder's top-layer features tends to lose low-level prosody,

transient events, and local time-frequency structure. To address this, we

adopt a DeepStack-inspired cross-layer injection module between the

encoder and the language model: in addition to the encoder's final-layer

output, features from earlier and intermediate layers are selected,

independently projected, and injected into the language model's early layers,

preserving multi-granularity information from low-level acoustic details to

high-level semantic abstractions.

This design is especially well-suited for music understanding, as it helps

retain rhythm, timbre, transients, and instrumental texture — information

that a single high-level representation cannot fully capture, yet is critical

for chord recognition, structural analysis, and nuanced musical description.

Time-Aware Representation

Time is a critical dimension in music understanding. To enhance explicit

temporal awareness, we adopt a time-marker insertion strategy during

pre-training: explicit time tokens are inserted between audio frame

representations at fixed time intervals to indicate temporal positions.

This design enables the model to learn "what happened when" within a unified

text generation framework, naturally supporting timestamped lyrics ASR,

beat / downbeat localisation, section boundary detection, and long-song

retrospective QA.

Building on the MOSS-Audio backbone, MOSS-Music is further enhanced through:

• continual pre-training on a large, diverse music corpus produced by

the data annotation and processing pipeline

MOSS-Music-Data-Pipeline,

with an emphasis on singing, lyrics, and full-song coverage;

• supervised fine-tuning (SFT) on music-centric instruction data covering

captioning, lyrics ASR, chord / key / structural analysis, and long-form

musical QA;

• additional reasoning tuning for the Thinking variant.

Released Models

| Model | Audio Encoder | LLM Backbone | Total Size | Hugging Face | ModelScope |

|---|---|---|---:|---|---|

| MOSS‑Music‑8B‑Instruct | MOSS-Audio-Encoder | Qwen3-8B | ~9.1B |  |  |

| MOSS‑Music‑8B‑Thinking | MOSS-Audio-Encoder | Qwen3-8B | ~9.1B |  |  |

Smaller (4B) variants and additional sizes may follow. Stay tuned!

Music Data Pipeline

The training data used by MOSS-Music is produced by an end-to-end pipeline

that goes from raw audio to chat-formatted training samples. That pipeline is

available at

MOSS-Music-Data-Pipeline,

which hosts duration detection, MIR feature extraction, song-structure

segmentation, lyrics ASR, metadata cleanup, and ALM-driven caption / query

generation with models such as Qwen3-Omni, MusicFlamingo, and other

audio-language models.

Evaluation

We evaluate MOSS-Music on a diverse suite of public music understanding

benchmarks. Key results:

• Music QA and understanding: MOSS-Music-8B-Instruct achieves 80.38

average accuracy across 8 public music QA benchmarks (excluding the

three NSynth note-recognition tracks), ranking first among all compared

models in our current evaluation set.

• Music captioning: In our preliminary GPT-5.4-as-a-Judge evaluation,

the MOSS-Music series leads both caption benchmarks, with

MOSS-Music-8B-Thinking reaching 4.53 on MusicCaps and

MOSS-Music-8B-Instruct reaching 4.58 on SDD.

• Lyrics ASR for singing voice: MOSS-Music-8B-Thinking achieves the

best average lyrics recognition error across MUSDB18, MIR-1K and

Opencpop (15.88% avg WER/CER), clearly ahead of all compared

audio-language baselines including Gemini-3.1-Pro-Preview,

MusicFlamingo and Qwen3-Omni. Detailed timestamped-ASR results will be

released in a later update.

• Chord transcription: MOSS-Music supports chord transcription, including

timestamped chord transcription for harmonic analysis, accompaniment

reference, and related downstream use cases. Detailed benchmark results will

be released in a later update.

Music QA & Understanding (Accuracy↑)

| Model | MMAU-music | MMAU-mini-music | MMAU-Pro-music | MMAR-music | MuChoMusic | Music-AVQA | NSynth (instrument) | NSynth (source) | NSynth (pitch) | GTZAN | Medley-Solos-DB | Avg |

|-----|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|

| MOSS‑Music‑8B‑Instruct | 79.33 | 80.78 | 71.02 | 59.70 | 89.39 | 76.78 | 86.55 | 61.07 | 86.94 | 93.59 | 92.42 | 80.38 |

| Gemini‑3.1‑Pro | 71.69 | 77.18 | 73.06 | 71.64 | 79.53 | 61.51 | 13.38 | 38.90 | 6.47 | 86.39 | 80.34 | 75.17 |

| MOSS‑Music‑8B‑Thinking | 74.09 | 77.78 | 67.98 | 50.25 | 82.90 | 68.90 | 56.17 | 57.48 | 77.83 | 84.78 | 87.42 | 74.26 |

| MusicFlamingo | 76.83 | 76.35 | 65.60 | 48.66 | 74.58 | 73.60 | 80.76 | 75.89 | 0.00 | 84.45 | 90.86 | 73.87 |

| Audio‑Flamingo‑Next | 72.39 | 72.07 | 61.64 | 45.27 | 75.62 | 62.94 | 86.40 | 66.73 | 0.05 | 77.68 | 91.47 | 69.89 |

| MiMo‑Audio‑7B‑Instruct | 66.36 | 72.97 | 66.50 | 45.77 | 75.40 | 57.05 | 25.01 | 1.49 | 4.86 | 65.67 | 93.81 | 67.94 |

| Step‑Audio‑R1 | 66.46 | 75.08 | 62.34 | 50.75 | 72.62 | 57.98 | 13.75 | 15.87 | 2.39 | 73.67 | 82.45 | 67.67 |

| Qwen3‑Omni | 65.76 | 68.77 | 66.27 | 48.54 | 78.77 | 56.05 | 30.92 | 44.30 | 28.08 | 80.15 | 69.65 | 66.75 |

| Kimi‑Audio‑7B‑Instruct | 47.95 | 52.25 | 59.10 | 45.27 | 70.18 | 68.90 | 6.01 | 0.81 | 3.88 | 39.54 | 71.98 | 56.90 |

Avg is computed over 8 public music QA benchmarks:

MMAU-music, MMAU-mini-music, MMAU-Pro-music, MMAR-music,

MuChoMusic, Music-AVQA, GTZAN, and Medley-Solos-DB.

We exclude the three NSynth tracks from the main average because they focus

on fine-grained isolated-note recognition, including instrument-family,

acoustic/electronic source, and exact pitch discrimination from short

single-note clips. Some compared audio-language models are not explicitly

designed for this note-level classification setting, so we report NSynth

separately for reference rather than mixing it into the headline average.

Music Captioning

We further report a preliminary GPT-5.4-as-a-Judge music captioning

comparison on MusicCaps and Song Describer Dataset (SDD). Scores are on a

1-5 scale across 9 dimensions: genre/style, mood/affect, tempo/rhythm,

instrumentation/timbre, vocals, melody/harmony, structure/form,

production/audio quality, and scene/use case.

• Overall caption quality: the MOSS-Music series remains strongest across

both caption benchmarks, with MOSS-Music-8B-Thinking reaching 4.53 on

MusicCaps and MOSS-Music-8B-Instruct reaching 4.58 on SDD.

• Stronger structural descriptions: MOSS-Music shows the clearest gains on

structure / form / progression, especially on SDD.

• Competitive baselines on instrumentation and scene semantics:

MusicFlamingo and Gemini-3.1-Pro remain competitive on

instrumentation/timbre, while Gemini-3.1-Pro is strongest on

scene / use case.

MusicCaps

| Model | Genre | Mood | Tempo | Instr. | Vocals | Melody/Harmony | Structure | Production | Scene | Avg |

|-----|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|

| MOSS‑Music‑8B‑Thinking | 4.78 | 4.69 | 4.62 | 4.40 | 4.46 | 4.40 | 4.86 | 4.35 | 4.18 | 4.53 |

| Gemini‑3.1‑Pro | 4.70 | 4.60 | 4.48 | 4.68 | 4.18 | 4.18 | 3.86 | 4.40 | 4.72 | 4.42 |

| MOSS‑Music‑8B‑Instruct | 4.60 | 4.52 | 4.46 | 4.02 | 4.30 | 4.38 | 4.78 | 4.20 | 3.96 | 4.36 |

| MusicFlamingo | 4.80 | 4.36 | 4.50 | 4.64 | 3.94 | 4.08 | 3.58 | 4.30 | 3.72 | 4.21 |

| Audio‑Flamingo‑Next | 4.34 | 4.56 | 4.08 | 4.30 | 4.18 | 3.78 | 3.66 | 4.04 | 3.92 | 4.10 |

| MiMo‑Audio‑7B‑Instruct | 4.02 | 4.20 | 4.46 | 4.28 | 4.36 | 3.62 | 3.30 | 4.08 | 3.50 | 3.98 |

| Step‑Audio‑R1 | 4.22 | 4.02 | 4.20 | 3.96 | 3.84 | 4.02 | 3.24 | 4.10 | 3.54 | 3.90 |

| Qwen3‑Omni | 4.58 | 4.50 | 4.26 | 3.62 | 3.64 | 3.48 | 2.98 | 4.18 | 4.42 | 3.96 |

| Kimi‑Audio‑7B‑Instruct | 3.98 | 3.92 | 4.32 | 3.88 | 4.48 | 3.28 | 2.72 | 3.72 | 3.24 | 3.73 |

Song Describer Dataset (SDD)

| Model | Genre | Mood | Tempo | Instr. | Vocals | Melody/Harmony | Structure | Production | Scene | Avg |

|-----|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|

| MOSS‑Music‑8B‑Instruct | 4.84 | 4.76 | 4.68 | 4.24 | 4.52 | 4.56 | 4.92 | 4.42 | 4.24 | 4.58 |

| Gemini‑3.1‑Pro | 4.72 | 4.64 | 4.52 | 4.72 | 4.22 | 4.24 | 3.94 | 4.46 | 4.82 | 4.48 |

| MOSS‑Music‑8B‑Thinking | 4.66 | 4.58 | 4.50 | 4.36 | 4.36 | 4.44 | 4.84 | 4.26 | 4.02 | 4.45 |

| MusicFlamingo | 4.82 | 4.40 | 4.52 | 4.70 | 3.98 | 4.14 | 3.66 | 4.36 | 3.80 | 4.26 |

| Audio‑Flamingo‑Next | 4.40 | 4.62 | 4.14 | 4.36 | 4.22 | 3.84 | 3.74 | 4.10 | 4.00 | 4.16 |

| MiMo‑Audio‑7B‑Instruct | 4.08 | 4.26 | 4.52 | 4.34 | 4.42 | 3.70 | 3.38 | 4.16 | 3.58 | 4.05 |

| Step‑Audio‑R1 | 4.30 | 4.10 | 4.26 | 4.02 | 3.92 | 4.10 | 3.32 | 4.18 | 3.62 | 3.98 |

| Qwen3‑Omni | 4.62 | 4.54 | 4.30 | 3.68 | 3.70 | 3.56 | 3.06 | 4.24 | 4.50 | 4.02 |

| Kimi‑Audio‑7B‑Instruct | 4.04 | 3.98 | 4.38 | 3.96 | 4.54 | 3.36 | 2.80 | 3.80 | 3.32 | 3.80 |

Lyrics ASR (WER / CER↓)

We further evaluate MOSS-Music on singing-voice lyrics ASR across three

representative benchmarks:

• MUSDB18 — English pop songs with backing tracks, scored with WER;
• MIR-1K — Chinese karaoke clips with background music, scored with CER;
• Opencpop — clean Mandarin studio singing, scored with CER.

Avg is the unweighted mean of the three dataset-level error rates.

| Model | MUSDB18 WER | MIR-1K CER | Opencpop CER | Avg |

|-----|---:|---:|---:|---:|

| MOSS‑Music‑8B‑Thinking | 29.19% | 15.84% | 2.60% | 15.88% |

| MOSS‑Music‑8B‑Instruct | 32.99% | 23.96% | 4.62% | 20.52% |

| Gemini‑3.1‑Pro‑Preview | 26.25% | 36.37% | 6.00% | 22.87% |

| MusicFlamingo | 23.41% | 38.98% | 18.73% | 27.04% |

| Qwen3‑Omni‑30B‑A3B‑Instruct | 62.67% | 20.48% | 2.26% | 28.47% |

| MiMo‑Audio‑7B‑Instruct | 94.16% | 23.34% | 6.77% | 41.42% |

| Kimi‑Audio‑7B‑Instruct | 97.53% | 25.83% | 4.90% | 42.75% |

| Step‑Audio‑R1 | 81.67% | 48.03% | 4.15% | 44.62% |

| Audio‑Flamingo‑Next | 94.93% | 55.63% | 12.47% | 54.34% |

MOSS-Music-8B-Thinking achieves the lowest average lyrics-ASR error

(15.88%) across these three datasets, with particular gains on

MIR-1K (Chinese karaoke with accompaniment) and Opencpop (clean Mandarin

singing). MOSS-Music also inherits the strong timestamp-aware ASR ability

from MOSS-Audio; detailed singing-timestamp ASR results will be added soon.

Chord Transcription

MOSS-Music supports chord transcription, including timestamped chord

transcription that tracks chord progression over time. This can be useful for

harmonic analysis, accompaniment reference, music education, and related use

cases. Detailed benchmark results will be added soon.

Quickstart

Environment Setup

We recommend Python 3.12 with a clean Conda environment. The commands below

are enough for local inference.

Recommended setup

git clone https://github.com/OpenMOSS/MOSS-Music.git
cd MOSS-Music

conda create -n moss-music python=3.12 -y
conda activate moss-music

conda install -c conda-forge "ffmpeg=7" -y
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime]"

Optional: FlashAttention 2

If your GPU supports FlashAttention 2, you can replace the last install

command with:

pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,flash-attn]"

SGLang Serving

[!IMPORTANT]

To achieve the best generation quality and fully leverage the model's capabilities, we

strongly recommend using SGLang Serving for inference.

See the full SGLang guide in moss_music_usage_guide.md.

Download the model first:

hf download OpenMOSS-Team/MOSS-Music-8B-Instruct --local-dir ./weights/MOSS-Music-8B-Instruct
hf download OpenMOSS-Team/MOSS-Music-8B-Thinking --local-dir ./weights/MOSS-Music-8B-Thinking

The shortest setup is:

cd sglang
pip install -e "python[all]"
pip install nvidia-cudnn-cu12==9.16.0.29
cd ..

sglang serve \
  --model-path ./weights/MOSS-Music-8B-Instruct \
  --trust-remote-code

You can replace ./weights/MOSS-Music-8B-Instruct with

./weights/MOSS-Music-8B-Thinking if needed.

If you use the default torch==2.9.1+cu128 runtime, installing

nvidia-cudnn-cu12==9.16.0.29 is recommended before starting sglang serve.

Gradio App

Start the Gradio demo with:

python app.py

The server address and port can be overridden via the

MOSS_MUSIC_SERVER_NAME and MOSS_MUSIC_SERVER_PORT environment variables,

and the default model ID via MOSS_MUSIC_MODEL_ID.

More Information

• MOSI.AI: https://mosi.cn
• OpenMOSS: https://www.open-moss.com
• MOSS-Audio (backbone): https://github.com/OpenMOSS/MOSS-Audio
• MOSS-Music Data Pipeline: https://github.com/wx9songs/MOSS-Music-Data-Pipeline

LICENSE

Models in MOSS-Music are licensed under the Apache License 2.0.

Citation

@misc{mossmusic2026,
      title={MOSS-Music Technical Report},
      author={OpenMOSS Team},
      year={2026},
      howpublished={\url{https://github.com/OpenMOSS/MOSS-Music}},
      note={GitHub repository}
}