Visual-ERM

Visual-ERM is a multimodal generative reward model for vision-to-code tasks.
It evaluates outputs directly in the rendered visual space and produces fine-grained, interpretable, and task-agnostic discrepancy feedback for structured visual reconstruction.

📄 Paper | 💻 GitHub | 📊 VC-RewardBench

Model Overview

Existing rewards for vision-to-code usually fall into two categories:

1. Text-based rewards such as edit distance or TEDS, which ignore important visual cues like layout, spacing, alignment, and style.
2. Vision embedding rewards such as DINO similarity, which are often coarse-grained and can be vulnerable to reward hacking.

Visual-ERM addresses this by directly comparing:

• the ground-truth image, and
• the rendered image produced from a model prediction,

and then generating structured discrepancy annotations that can be converted into reward signals or used for reflection-based refinement.

What this model does

Visual-ERM is designed to judge whether a predicted result is visually equivalent to the target.

Given a pair of images, it can identify discrepancies such as:

• category
• severity
• location
• description

This makes Visual-ERM useful not only as a reward model for RL, but also as a visual critic for test-time reflection and revision.

Supported Tasks

Visual-ERM is designed for structured visual reconstruction tasks, including:

• Chart-to-Code
• Table-to-Markdown
• SVG-to-Code

Key Features

• Visual-space reward modeling
  Evaluates predictions in rendered visual space instead of relying only on text matching or coarse embedding similarity.

• Fine-grained and interpretable feedback
  Produces structured discrepancy annotations rather than a single black-box score.

• Task-agnostic reward supervision
  A unified reward model that generalizes across multiple vision-to-code tasks.

• Useful for both training and inference
  Can be used as a reward model in RL and as a visual critic during test-time refinement.

VC-RewardBench

We also release VisualCritic-RewardBench (VC-RewardBench), a benchmark for evaluating fine-grained image-to-image discrepancy judgment on structured visual data.

Benchmark Features

• Covers charts, tables, and SVGs
• Contains 1,335 carefully curated instances
• Each instance includes:
  • a ground-truth image
  • a corrupted / rendered counterpart
  • fine-grained discrepancy annotations

Dataset link:
https://huggingface.co/datasets/internlm/VC-RewardBench

How to Use

Visual-ERM is fine-tuned from Qwen/Qwen3-VL-8B-Instruct and follows the same multimodal interface.

Input

Visual-ERM takes as input:

• a reference / ground-truth image
• a rendered prediction image
• a prompt asking the model to identify fine-grained visual discrepancies

Output

The model outputs structured discrepancy annotations, which can then be:

• converted into a scalar reward for RL
• used as feedback for reflection-and-revision
• evaluated directly on VC-RewardBench

A typical output format is:

{
  "errors": [
    {
      "category": "structure_error",
      "severity": 3,
      "location": "legend area",
      "description": "The legend is placed outside the plot area in the prediction."
    },
    {
      "category": "style_error",
      "severity": 2,
      "location": "bar colors",
      "description": "The colors differ from those in the reference image."
    }
  ]
}

Inference / Evaluation / RL

For full inference scripts, RL training pipelines, evaluation code, and prompt templates, please refer to the official repository:

https://github.com/InternLM/Visual-ERM

Intended Use

Visual-ERM is intended for:

• reward modeling in vision-to-code RL pipelines
• visual discrepancy judgment between target and predicted renderings
• reflection-based refinement at inference time
• research on visual reward modeling and multimodal RL

Citation

If you find this model useful, please consider citing:

@article{liu2026visual,
  title={Visual-ERM: Reward Modeling for Visual Equivalence},
  author={Liu, Ziyu and Ding, Shengyuan and Fang, Xinyu and Dai, Xuanlang and Yang, Penghui and Liang, Jianze and Wang, Jiaqi and Chen, Kai and Lin, Dahua and Zang, Yuhang},
  journal={arXiv preprint arXiv:2603.13224},
  year={2026}
}

Contact

If you are interested in visual reward modeling, vision-to-code, or reinforcement learning for multimodal models, feel free to reach out.