Hunmin_vlm_235b_v0.11_merged_cua

1. Overview

Author: 임성준 (Sungjun Lim) [LinkedIn]
Role: Project Lead & Primary Researcher
Base model: Qwen/Qwen3-VL-235B-A22B-Instruct (MoE, 235B total / ~22B active)
Adapter: LoRA with layer-wise rank allocation and adaptive learning rate scheduling
Focus: Enhancing grounding, UI understanding, and end-to-end computer-use task execution
License: Apache 2.0 (model weights & code)

This model is fine-tuned on Qwen3-VL-235B-A22B-Instruct to enhance computer use and browser use performance, while preserving general Korean VLM/LLM capabilities.

---

2. Evaluation Results

2-1. Visual Agent (Computer Use & Browser Use)

The final performance improvements were verified to be statistically significant based on six independent evaluation runs (p-value < 0.0001).

i. OSWorld (Simulation Benchmark)

• An average performance improvement of approximately 2% was observed.
• When comparing the best runs, the success rate increased from 31.85% (baseline) to 34.9% (LoRA-tuned).

• In particular, significant performance improvements were observed in the vlc, chrome, and libreoffice_writer domains.

ii. OSWorld-G (UI Grounding Benchmark)

• An average improvement of approximately 2% was observed in grounding performance, with the largest gain (+3.92%) occurring in fine-grained manipulation.

iii. ScreenSpot Pro (UI Grounding Benchmark)

• These results were obtained with max pixel = 2,073,600, and may therefore differ from values reported under other evaluation settings.

---

2-2. Korean General Capability

This training focuses on multimodal grounding and agent optimization, and the results indicate that general-purpose multimodal and text capabilities are preserved at their original performance levels.

i. Multimodal Benchmarks

ii. Language & Reasoning Benchmarks

---

3. Curated Training Data (≈1,000 High-Quality Samples)

Curated multimodal instruction-tuning data focused on UI grounding, layout understanding, icon recognition, and interaction trajectories:

• Component & description grounding (~450 samples)
  Rule-based + generated datasets covering documents, slides, scrolling, snap icons, component libraries

• Icon grounding (~200 samples)
  Icon-specific vision-language alignment data

• Layout understanding (~200 samples)
  Layout200k, Layout400k (Claude-augmented), OS layout datasets

• UI / Interaction trajectories (~150 samples)
  SeeClick, GUIEnv, WebUI, OmniAct, Mind2Web, AndroidControl, RicoSCA, etc.

---

4. Future Work & Extensions

While this release empirically demonstrates the effectiveness of lightweight LoRA adaptation, it represents only an initial step toward fully optimized multimodal computer-use agents. Several key directions are planned for future extensions:

• Full-Parameter Optimization
  The current model applies LoRA-based fine-tuning. Future work will explore full-parameter fine-tuning for further improvements.

• Reinforcement Learning
  To move beyond supervised fine-tuning, we plan to incorporate reinforcement learning–based optimization methods. These approaches will directly reward successful long-horizon task completion and robustness under real-world UI uncertainty.

• Long-Horizon Trajectory Data Synthesis
  Complex computer-use tasks often require reasoning over extended action sequences across multiple applications. To address this, we will develop a scalable data synthesis pipeline that generates long-horizon, multi-step interaction trajectories.

• Expanded Evaluation Ecosystem
  Additional benchmarks will be included to provide a more comprehensive evaluation of real-world performance.

Together, these extensions aim to transition the model from a grounding-focused PoC to a strong multimodal agent.