SegFormer-B0 — Road Scene Segmentation (7 Game-Asset Classes)

A SegFormer-B0 model fine-tuned on Cityscapes for 7-class semantic segmentation of road scenes. This model uses a purpose-built taxonomy where every class maps directly to a game element — road texture, sky backdrop, tree sprites, building silhouettes, etc.

7-Class Game Taxonomy

Key design decisions

• Road ↔ sidewalk kept separate so road color sampling produces pure grey asphalt without contamination from sidewalk / terrain tones.
• Terrain grouped with sidewalk, not vegetation — ground-level grass strips serve the same game function as sidewalk.
• All building-like structures merged (building, wall, fence) — the game treats them identically as background geometry.
• All vehicle types merged — the game treats every vehicle as a road obstacle.
• Roadside objects (poles, traffic lights, signs, persons, riders) are grouped into a single thin-element class with fallback stock sprites when extraction quality is low.

Usage

from transformers import SegformerImageProcessor, SegformerForSemanticSegmentation
from PIL import Image
import torch
import torch.nn.functional as F

processor = SegformerImageProcessor.from_pretrained("Marco333/segformer-b0-road-scene-7class")
model = SegformerForSemanticSegmentation.from_pretrained("Marco333/segformer-b0-road-scene-7class")

image = Image.open("road_photo.jpg")
inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

# Upsample logits to original image size
mask = F.interpolate(
    outputs.logits,
    size=image.size[::-1],  # (H, W)
    mode="bilinear",
    align_corners=False,
).argmax(dim=1)[0]

# mask values: 0=road, 1=sidewalk, 2=building, 3=vegetation, 4=sky, 5=vehicle, 6=roadside_object

Training Details

Dataset

Chris1/cityscapes_segmentation — urban street scenes from 50 European cities.

The original Cityscapes masks use label IDs 0–33 stored as 3-channel RGB images. A custom preprocessing pipeline extracts channel 0 and applies a 256-element lookup table to remap all 34 Cityscapes classes into the 7-class game taxonomy in a single vectorized operation (unmapped classes → 255 = ignore).

Hyperparameters

Training Curve

The model converges quickly thanks to transfer learning — the pretrained encoder already understands road scene features; only the 7-class decoder head is learned from scratch.

Implementation Notes

Several non-obvious flags are required for correct training:

• ignore_mismatched_sizes=True — the pretrained decoder head has a different number of output classes.
• remove_unused_columns=False — prevents the Trainer from dropping image data columns.
• label_names=["labels"] — tells the Trainer which key holds the segmentation targets.
• do_reduce_labels=False — Cityscapes labels don't need ADE20K-style background subtraction.
• Logit upsampling in compute_metrics — SegFormer outputs at ¼ resolution; logits must be upsampled before comparison with ground-truth masks.

Intended Use

This model is designed for a game asset extraction pipeline where a user uploads a road photograph and the runtime transforms it into game elements:

1. Road shape — fit road boundaries from the road mask; derive perspective and horizon.
2. Color palette — sample dominant colors from each masked region of the original image.
3. Sky — crop the sky band directly using the sky mask.
4. Tree sprites — blob detection on the vegetation mask; crop with alpha transparency.
5. Building silhouettes — extract from the building mask for background geometry.
6. Fallback system — when extraction quality is poor for any element, use palette-matched stock assets.

Limitations

• Trained exclusively on European urban street scenes (Cityscapes). Performance may degrade on rural roads, highways without sidewalks, non-European road styles, or indoor scenes.
• roadside_object class (52.7 % IoU) is the weakest — thin elements like poles and signs are inherently difficult at 512×512 resolution. The intended runtime uses fallback sprites for this class.
• Not suitable for safety-critical autonomous driving — the merged taxonomy intentionally discards distinctions (truck vs. car, wall vs. fence) that matter for driving but not for game art.

Citation

Please cite this model if you use it:

@misc{corbetta_segformer_road_scene_7class_2026,
  author       = {Marco Corbetta},
  title        = {segformer-b0-road-scene-7class: SegFormer-B0 fine-tuned on Cityscapes for 7-class game-asset segmentation},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Marco333/segformer-b0-road-scene-7class}}
}

SegFormer:

@inproceedings{xie2021segformer,
  title={SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers},
  author={Xie, Enze and Wang, Wenhai and Yu, Zhiding and Anandkumar, Anima and Alvarez, Jose M and Luo, Ping},
  booktitle={NeurIPS},
  year={2021}
}