Xiaomi-Lens-0

Xiaomi-Lens-0 is a 4B multimodal language model built on Qwen3.5-4B. It adds a lightweight pixel-mask decoder and is trained with joint language and mask supervision. Each generated <|mask|> token identifies an image region that the decoder turns into a segmentation mask.

The model supports referring expression comprehension and segmentation, multi-instance grounding, reasoning segmentation, embodied referring, and fine-grained visual perception. It can produce masks in its final answer or during reasoning.

This repository contains the model weights, configuration, tokenizer, processor, and chat template. Inference uses the XiaomiLensQwen3_5ForConditionalGeneration class from the accompanying Xiaomi-Lens-0 code repository. Install that package to load the model and decode masks.

📰 News

• The Xiaomi-Lens-0 online demo is now live on Hugging Face Spaces.
• The Xiaomi-Lens-0 Technical Report is now publicly available on arXiv.

🎬 Overview video

Watch on the project page · Download the MP4

🖼️ Task showcases

Recorded model outputs across six task domains. Each task includes four examples from the GitHub project.

Referring Expression Comprehension

Referring Expression Segmentation

Multi-Instance Grounding

Reasoning Segmentation

Embodied Referring

Fine-Grained Perception

Quick start

Inference requires Python 3.10 or later and a CUDA GPU with bfloat16 and FlashAttention-2 support. From a checkout of the Xiaomi-Lens-0 inference repository, install the dependencies and package:

pip install -r requirements.txt
pip install -e .

Run the inference script with the downloaded checkpoint directory and an image:

python infer.py \
  --checkpoint /path/to/Xiaomi-Lens-0 \
  --image /path/to/image.jpg \
  --text 'Locate all instances in the image that match the description "the cat on the left". Report bbox coordinates and masks.' \
  --output result.jpg --draw-bbox

The script prints the generated text and saves an image with the predicted masks and bounding boxes. --text accepts a complete instruction; the checkpoint's chat template supplies the image tokens and assistant prefix.

Each grounded instance is represented as <bbox>[x1, y1, x2, y2]</bbox><mask><|mask|></mask>, with box coordinates normalized to 0–1000. Mask decoding uses the contextual state of each <|mask|> token and the vision encoder's features. The inference script excludes masks inside <think> from the final-answer overlay.

Serving with vLLM

Use a separate environment for vLLM serving. From the inference repository, install the serving dependencies:

pip install -e ".[vllm]"

Run text generation and mask decoding in separate terminals:

# Terminal 1: text generation on port 8000
bash src/xiaomi_lens/integrations/vllm/run_vllm.sh /path/to/Xiaomi-Lens-0

# Terminal 2: mask decoding on port 8001
bash src/xiaomi_lens/integrations/vllm/run_vllm_mask.sh /path/to/Xiaomi-Lens-0

# Terminal 3: query both services
python -m xiaomi_lens.integrations.vllm.client \
  --image /path/to/image.jpg --query "the cat on the left" --output result.png

The client must run on the same machine as the services because it sends the image path to the mask server. The launchers prepare temporary checkpoint views for vLLM. If you override MAX_PIXELS, use the same value for both services.

Training and evaluation

Training first pretrains the mask decoder, then fine-tunes the full model with language and pixel-level supervision. The inference repository's README provides the benchmark tables and recorded examples across the six capability domains.

LensBench

Evaluate instruction-conditioned segmentation across general, aerial, driving, and panoramic imagery with LensBench.

The original baseline results are reported in Table 7 in Section 4.3 of the Xiaomi-Lens-0 Technical Report (arXiv v1). The table below also includes the subsequently added GPT-6-Astra results and is synchronized with the project results.

Marginal results pool single and multiple targets. Each cell reports cIoU / Prec@0.5 (%, higher is better). Bold marks the best score, and underlining marks the second-best score for each metric and column, including ties.

GPT-6-Astra leads cIoU on explicit and implicit instructions and in aerial, driving, and panoramic imagery. Qwen3.8-Max leads cIoU in general imagery (74.2). Xiaomi-Lens-0 ranks second on both explicit-instruction metrics and on cIoU in aerial, driving, and panoramic imagery, and leads aerial Prec@0.5 (79.6). Implicit instructions remain challenging: cIoU drops from 71.4 to 54.7, while Prec@0.5 drops from 85.0 to 70.0. The gap is largest for multiple targets in panoramic scenes, where cIoU falls from 71.8 to 48.9 and Prec@0.5 from 81.3 to 48.7.

Predictions can miss objects or produce inaccurate boxes and masks, particularly for small, occluded, or ambiguous regions. Masks are decoded on the image canvas produced by the processor; preserve the image resizing and coordinate mapping when consuming them. The supplied inference path supports images.

License and attribution

The model is released under the Apache License 2.0.

Xiaomi-Lens-0 derives from Qwen3.5-4B, whose weights are also licensed under Apache 2.0. Xiaomi-Lens-0 adds the pixel-mask decoder and fine-tunes the model for grounded language and segmentation tasks.

• Qwen3.5-4B: Copyright 2026 Alibaba Cloud.
• Xiaomi-Lens-0 modifications: Copyright 2026 Xiaomi Corporation.

Citation

@misc{xiaomilens0,
  title         = {Xiaomi-Lens-0 Technical Report},
  author        = {Pei Fu and Yike Liu and Yue Sun and Hanmo Wang and
                   Xiuwen Xi and Ruoceng Zhang and Shaojie Zhang and Tingwei Huang and
                   Chao Jiang and Hang Li and Bingrui Liu and Zhenbo Luo and
                   Heng Qu and Changjian Xu and Ming Xu and Jian Luan},
  year          = {2026},
  eprint        = {2610.23279},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2610.23279}
}