GRiP-7B: Guiding the Inner Eye

Arxiv | Huggingface

Overview

This repository contains the official model checkpoints of GRiP (Guided Reasoning and Perception), a novel visual grounded reasoning model developed by Basic Algorithm Center, Platform and Content Group, Tencent.

Models capable of "thinking with images" represent a major leap in multimodal AI. GRiP is designed to cultivate robust and flexible visual grounded reasoning by explicitly guiding the model's perceptual focus and logical pathways. Initialized from Qwen2.5-VL-7B-Instruct, GRiP employs a two-stage training framework:

1. Bootstrapping: Structured instruction tuning to teach the syntax of grounded reasoning.
2. Policy Refinement: A cognitive-enhanced Reinforcement Learning (RL) stage featuring novel reward mechanisms.

GRiP achieves state-of-the-art results among open-source models on challenging benchmarks like TreeBench, V* Bench, and HR-Bench, demonstrating superior capability in complex visual reasoning.

Methodology

The core of GRiP lies in its Policy Refinement stage, which addresses the "Coarse Reward Problem" in existing RL methods. We introduce a multi-faceted reward architecture:

$$ R_{\text{total}} = R_{\text{acc}} + R_{\text{fmt}} + R_{\text{sw-IoU}} + R_{\text{MHR}} $$

Where:

• Salience-Weighted IoU Reward ($R_{\text{sw-IoU}}$): Incentivizes the model to prioritize mission-critical objects over trivial distractors. It weights the recall component by an object's salience score $s_k$: $$ R_{\text{recall}} = \frac{1}{\sum s_k} \sum_{k=1}^{M} s_k \cdot \max_{i} \text{IoU}(p_i, g_k) $$

• Multi-Heuristic Reward ($R_{\text{MHR}}$): Encourages cognitive flexibility by rewarding diverse valid reasoning pathways (e.g., Bottom-Up, Top-Down, Deductive Verification). The model is rewarded based on similarity to the best-matching reference trajectory: $$ R_{\text{MHR}} = \max_{j \in {1,2,3}} \text{sim}(\tau_{\text{gen}}, \tau_{\text{ref}}^j) $$

Performance

TreeBench Evaluation

TreeBench is a highly challenging benchmark for fine-grained perception and multi-step reasoning. GRiP significantly outperforms its base model and other open-source competitors.

Generalization on V* Bench and HR-Bench

GRiP demonstrates strong generalization capabilities on attribute recognition, spatial understanding, and high-resolution reasoning.

Train and Inference

Please refer to our Huggingface Repository for training and inference codes.

Training Details

• Hardware: 8 $\times$ NVIDIA H20 (96GB) GPUs.
• Frameworks: LLaMA-Factory for SFT, EasyRL for RL training.
• Optimization: AdamW optimizer, GRPO algorithm for Policy Refinement.

Acknowledgements

Our work is built upon the excellent Qwen2.5-VL. We also thank the developers of LLaMA-Factory and EasyRL for their efficient training frameworks.

Citation

If you find our work helpful, please cite:

@article{wei2025grip,
  title={Guiding the Inner Eye: A Framework for Hierarchical and Flexible Visual Grounded Reasoning},
  author={Wei, Zhaoyang and Ding, Wenchao and Hao, Yanchao and Chen, Xi},
  journal={arXiv preprint},
  year={2025}
}