MOSS-VL-Instruct-0708

MOSS-VL is an open vision-language model family from OpenMOSS, supporting image understanding, long-video understanding, and realtime streaming interaction.

Technical Report: https://arxiv.org/pdf/2608.15045

Introduction

MOSS-VL-Instruct-0708 is the instruction-tuned checkpoint of the MOSS-VL 0708 release, part of the OpenMOSS ecosystem for open visual understanding.

Built on top of MOSS-VL-Base-0708 through supervised fine-tuning (SFT), this checkpoint is designed as a high-performance offline multimodal model. It supports image understanding, OCR, document parsing, visual reasoning, instruction following, and video understanding, with particular strength in long-form video comprehension, temporal reasoning, action recognition, and fine-grained event localization.

The 0708 release keeps the MOSS-VL cross-attention design and a 256K text context window while refreshing the data and instruction-tuning recipe for stronger offline multimodal usage.

Highlights

Strong video understanding: designed for long videos, temporal reasoning, action recognition, and second-level event localization.
General multimodal perception: supports image understanding, fine-grained visual recognition, OCR, and document analysis.
Reliable instruction following: SFT aligns the base checkpoint with user instructions across image, video, and text tasks.
Open model family: released together with MOSS-VL-Base-0708 for continued pretraining, fine-tuning, and applied research.

Model Architecture

MOSS-VL-Instruct-0708 adopts a cross-attention-based vision-language architecture that decouples visual encoding from language reasoning. The model processes images, videos, and text in a unified pipeline and uses cross-attention layers to connect language tokens with visual representations.

Key configuration details:

| Item | Value |

| --- | --- |

| Parameters | 11B |

| Tensor type | BF16 |

| Context length | 256K |

| Vision patch size | 16 |

| Temporal patch size | 1 |

| Default video FPS | 1.0 |

| Default max video frames | 256 |

Absolute Timestamps

For video inputs, MOSS-VL injects absolute timestamps alongside sampled frames. This helps the model reason about event order, duration, pacing, and temporal localization instead of relying only on frame order.

Cross-attention RoPE (XRoPE)

MOSS-VL uses Cross-attention Rotary Position Embedding (XRoPE), which maps text tokens and visual patches into a unified three-dimensional coordinate space defined by Time (t), Height (h), and Width (w). This gives the model a consistent positional representation for image and video reasoning.

Model Performance

MOSS-VL-Instruct-0708 is intended for offline multimodal evaluation across visual perception, multimodal reasoning, OCR/document understanding, and video understanding. Detailed benchmark tables for the 0708 release will be maintained in the MOSS-VL project resources.

For the previous public checkpoint, see MOSS-VL-Instruct-0408.

Quickstart

Installation

Clone the MOSS-VL repository and install the project requirements:

git clone https://github.com/OpenMOSS/MOSS-VL.git
cd MOSS-VL
conda create -n moss_vl python=3.12 pip -y
conda activate moss_vl
pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt

Load Model

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

checkpoint = "OpenMOSS-Team/MOSS-VL-Instruct-0708"

processor = AutoProcessor.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    frame_extract_num_threads=1,
)
model = AutoModelForCausalLM.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)

Run Inference

Single-image Inference

image_path = "data/example_image.jpg"
prompt = "Describe this image."

text = model.offline_image_generate(
    processor,
    prompt=prompt,
    image=image_path,
    shortest_edge=4096,
    longest_edge=16777216,
    multi_image_max_pixels=201326592,
    patch_size=16,
    temporal_patch_size=1,
    merge_size=2,
    image_mean=[0.5, 0.5, 0.5],
    image_std=[0.5, 0.5, 0.5],
    max_new_tokens=256,
    temperature=1.0,
    top_k=50,
    top_p=1.0,
    repetition_penalty=1.0,
    do_sample=False,
    vision_chunked_length=64,
)

print(text)

Single-video Inference

video_path = "data/example_video.mp4"
prompt = "Describe this video."

text = model.offline_video_generate(
    processor,
    prompt=prompt,
    video=video_path,
    shortest_edge=4096,
    longest_edge=16777216,
    video_max_pixels=201326592,
    patch_size=16,
    temporal_patch_size=1,
    merge_size=2,
    video_fps=1.0,
    min_frames=1,
    max_frames=256,
    num_extract_threads=4,
    image_mean=[0.5, 0.5, 0.5],
    image_std=[0.5, 0.5, 0.5],
    max_new_tokens=256,
    temperature=1.0,
    top_k=50,
    top_p=1.0,
    repetition_penalty=1.0,
    do_sample=False,
    vision_chunked_length=64,
)

print(text)

Batched Offline Inference

offline_batch_generate accepts independent image/video/text queries. Queries in the same batch should share the same media_kwargs and generate_kwargs.

queries = [
    {
        "prompt": "Describe sample A.",
        "images": [],
        "videos": ["data/sample_a.mp4"],
        "media_kwargs": {
            "video_fps": 1.0,
            "min_frames": 8,
            "max_frames": 256,
        },
        "generate_kwargs": {
            "temperature": 1.0,
            "top_k": 50,
            "top_p": 1.0,
            "max_new_tokens": 256,
            "repetition_penalty": 1.0,
            "do_sample": False,
        },
    },
    {
        "prompt": "Describe sample B.",
        "images": [],
        "videos": ["data/sample_b.mp4"],
        "media_kwargs": {
            "video_fps": 1.0,
            "min_frames": 8,
            "max_frames": 256,
        },
        "generate_kwargs": {
            "temperature": 1.0,
            "top_k": 50,
            "top_p": 1.0,
            "max_new_tokens": 256,
            "repetition_penalty": 1.0,
            "do_sample": False,
        },
    },
]

with torch.no_grad():
    result = model.offline_batch_generate(
        processor,
        queries,
        vision_chunked_length=64,
    )

texts = [item["text"] for item in result["results"]]
print(texts)

Related Checkpoints

| --- | ---: | ---: | --- | --- |

Limitations and Future Work

MOSS-VL-Instruct-0708 is optimized for general offline multimodal understanding. Very dense videos, highly specialized domains, precise small-text OCR, and tasks requiring strict numerical reasoning may still require task-specific prompting, sampling choices, or fine-tuning.

We are continuing to improve mathematical reasoning, code reasoning, RL post-training, and broader task-specific evaluations for future MOSS-VL releases.

Citation

@misc{mossvl,
  title         = {MOSS-VL Technical Report},
  author        = {Wang, Pengyu and Tan, Chenkun and Zhou, Shaojun and Zhou, Qirui and Chen, Yanxin and He, Xingyang and Zeng, Huazheng and Cheng, Jijun and Wang, Chenghao and Qian, Xiaomeng and Wang, Pengfei and Huang, Zhan and Gao, Shanqing and Huang, Wei and Cao, Longjun and Ran, Wu and Liu, Jie and Zhu, Changtai and Wang, Hongkai and Tian, Yixian and Liu, Chenghao and Ye, Zhen and Wang, Xinghao and Jiang, Botian and Feng, Guoguo and Fei, Zhaoye and Li, Ruixiao and Chen, Mingshu and Gao, Yang and Cheng, Qinyuan and Li, Shimin and Qiu, Xipeng},
  year          = {2026},
  eprint        = {2608.15045},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2608.15045}
}

@misc{mossvideopreview,
  title         = {{MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention}},
  author        = {Pengyu Wang and Chenkun Tan and Shaojun Zhou and Wei Huang and Qirui Zhou and Zhan Huang and Zhen Ye and Jijun Cheng and Xiaomeng Qian and Yanxin Chen and Xingyang He and Huazheng Zeng and Chenghao Wang and Pengfei Wang and Hongkai Wang and Shanqing Gao and Yixian Tian and Chenghao Liu and Xinghao Wang and Botian Jiang and Xipeng Qiu},
  year          = {2026},
  eprint        = {2606.07639},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2606.07639}
}