WaferSAGE-RL: Rubric-Guided Qwen3-VL for Wafer Map Defect Analysis

This repository contains a rubric-guided reinforcement learning model for wafer map visual question answering.

It is part of WaferSAGE, a framework for wafer defect analysis via synthetic data generation, structured rubric evaluation, and domain-specific post-training of small vision-language models.

Historical note: this model was originally released under the WMVLM-RL naming convention. It is now part of the WaferSAGE model family.

Model Summary

WaferSAGE-RL is initialized from a WaferSAGE supervised fine-tuned Qwen3-VL model and further optimized with curriculum-based reinforcement learning using rubric-aligned rewards.

The model is designed for semiconductor wafer map understanding, especially:

• Defect type identification
• Spatial distribution analysis
• Morphological description
• Multi-modal defect pattern interpretation
• Root-cause candidate generation
• Hallucination reduction through rubric-based penalties

Why Reinforcement Learning?

Supervised fine-tuning helps the model learn wafermap-specific language and response structure. However, SFT alone may still produce:

• Overly generic descriptions
• Missing key defect locations
• Hallucinated clock positions or wafer zones
• Incorrect morphology terms
• Plausible but unsupported root-cause explanations

WaferSAGE-RL uses structured rubrics as reward signals. These rubrics include both positive criteria and negative criteria:

must-hit:
  Terms or concepts the answer should include.

must-avoid:
  Incorrect locations, defect types, or morphology terms that should be penalized.

This encourages the model to cover important visual evidence while avoiding unsupported or hallucinated statements.

Post-training Pipeline

WaferSAGE SFT model
    ↓
Rubric-augmented VQA data
    ↓
Curriculum ordering by difficulty
    ↓
Multiple sampled completions per prompt
    ↓
Rubric-based reward scoring
    ↓
GSPO / GRPO-style policy optimization
    ↓
WaferSAGE-RL model

Curriculum Learning

The RL dataset combines previously seen SFT-style examples and additional rubric-augmented examples. Training examples are ordered from easier to harder samples, so the model first stabilizes on simpler recognition and localization tasks before moving to more complex multi-defect reasoning and root-cause hypothesis questions.

Example difficulty categories:

• Easy: direct defect type or location questions
• Medium: morphology and distribution questions
• Hard: multi-modal defect interpretation and root-cause hypothesis questions

Reward Design

The reward is based on rubric matching across major dimensions:

Root-cause rewards should be interpreted carefully: wafer maps alone cannot prove true process root cause. This dimension is intended to encourage plausible candidate explanations, not definitive diagnosis.

Usage with Transformers

from transformers import pipeline

pipe = pipeline(
    "image-text-to-text",
    model="Niraya666/Qwen3-VL-4B-Instruct-WMVLM-RL-0213"
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "path_or_url_to_wafermap.png"},
            {"type": "text", "text": "Describe the wafer map defect pattern and possible root-cause hypotheses."}
        ]
    }
]

output = pipe(text=messages, max_new_tokens=512)
print(output)

Usage with vLLM

vllm serve "Niraya666/Qwen3-VL-4B-Instruct-WMVLM-RL-0213"

Then call the OpenAI-compatible endpoint:

curl -X POST "http://localhost:8000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  --data '{
    "model": "Niraya666/Qwen3-VL-4B-Instruct-WMVLM-RL-0213",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Analyze this wafer map. Describe the spatial distribution, morphology, and possible root-cause hypotheses."
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "path_or_url_to_wafermap.png"
            }
          }
        ]
      }
    ],
    "max_tokens": 512
  }'

Suggested Prompt Template

For best results, ask the model to separate visual evidence from root-cause hypotheses:

Analyze this wafer map.

Please answer in three parts:
1. Spatial distribution: where are the defective dies located?
2. Morphology: what pattern, shape, density, or structure is visible?
3. Root-cause candidates: what process or equipment issues could plausibly explain this pattern?

Do not claim a definitive root cause unless there is enough process context.

Evaluation

WaferSAGE-RL was evaluated using a rubric-based wafermap VQA benchmark and LLM-as-a-Judge scoring.

The benchmark evaluates:

• Spatial understanding
• Morphological understanding
• Root-cause candidate quality
• Defect type identification
• Hallucination avoidance

In the WaferSAGE paper, the 4B Qwen3-VL RL model achieved a 6.493 LLM-Judge score, closely approaching Gemini-3-Flash at 7.149, while supporting fully local deployment.

Strengths

• Domain-adapted to wafer map visual understanding.
• Better at wafermap-specific terminology than generic VLMs.
• Supports local deployment for privacy-sensitive semiconductor environments.
• Uses rubric-guided RL to improve spatial and morphological answer quality.
• Useful for research prototypes and engineering assistant workflows.

Limitations

This model has important limitations:

• It does not perform definitive root-cause diagnosis.
• Root-cause outputs are candidate hypotheses only.
• It does not use lot history, tool/chamber logs, recipe parameters, metrology, inline inspection, or CP/FT data.
• It may still hallucinate plausible but unsupported process causes.
• It may overfit to synthetic rubric style and public wafermap data distributions.
• It should not be used for automated process control or high-cost manufacturing decisions without expert review.

Recommended Use

Recommended:

• Wafer map VQA research
• Industrial VLM evaluation
• Semiconductor AI assistant prototypes
• Local on-premise proof-of-concept deployments
• Synthetic-data and rubric-reward research

Not recommended:

• Final process root-cause diagnosis
• Production yield disposition without human review
• Automated manufacturing control
• Safety-critical or high-cost decisions without validation

Related Resources

• WaferSAGE paper: arXiv:2604.27629
• SFT starting model: Niraya666/Qwen3-4B-wmvlm-260204
• WaferSAGE VQA dataset: Niraya666/wafermap-vqa-2602
• Rubric datasets: Niraya666/wafermap-vqa-with-rubrics-2602, Niraya666/WaferSAGE-Wafermap-VQA-Dataset

Citation

If you use this model, please cite:

@misc{xu2026wafersage,
  title        = {WaferSAGE: Large Language Model-Powered Wafer Defect Analysis via Synthetic Data Generation and Rubric-Guided Reinforcement Learning},
  author       = {Ke Xu and Zhongyuan Lian},
  year         = {2026},
  eprint       = {2604.27629},
  archivePrefix= {arXiv},
  primaryClass = {cs.AI}
}