Ship direction — a U-Net that reads heading through heavy occlusion

Estimates the heading of a ship icon in an 80x80 top-down game frame where the ship may be almost entirely hidden behind portraits, village-name text, or map markers, and reconstructs the whole ship at that heading. 0 deg = up (north), clockwise.

The network labels every pixel as a ship part; a rigid geometric fit then turns those part masks into an angle, refusing any pose that would place the hull on open water.

The bottom two rows are the point: with a single part visible — only a stern (t082), only a bow (t083) — the geometry still pins the heading.

Two stages

frame -> SegNet (U-Net) -> per-pixel part labels -> part_pose (rigid fit) -> heading

The checkpoint is only half the system. SegNet outputs (6, 80, 80) class probabilities (bg, bow, hull, stern, sail_l, sail_r) and contains no notion of an angle. part_pose (plain NumPy/FFT, in estimator.py) searches all rotations for the single rigid pose that best explains those masks, scoring three terms:

The last term is what keeps reconstructions physically consistent: a ship may hide under an occluder, but not float on open water where it would have been seen.

Usage

import numpy as np
from PIL import Image
from huggingface_hub import hf_hub_download
from estimator import ShipHeading   # this repo's estimator.py

REPO = "fortunatetumbleweed/ship-direction"
est = ShipHeading(
    model_path=hf_hub_download(REPO, "model.safetensors"),
    canonical_path=hf_hub_download(REPO, "canonical.npz"),
)

img = np.asarray(Image.open("frame.png").convert("RGB"))   # uint8 (80,80,3), RGB
r = est(img)
r.heading      # float degrees, 0 = up/north, clockwise
r.parts_seen   # e.g. ['bow']
r.labels       # (80,80) uint8 part ids
recon = est.reconstruct(img, r)   # (80,80,3) uint8, whole ship painted in

Requires torch, numpy, pillow, safetensors.

Input contract

A wrong shape raises; these fail silently:

• RGB channel order (OpenCV gives BGR — convert first)
• 80x80 uint8, the raw frame; normalization (x/255 - 0.5) happens inside
• ship at the game's fixed scale, roughly centered

Results

On 21 labelled real frames:

Speed on an M-series CPU: ~60 ms init, then ~39 ms/frame (10 ms network + 29 ms pose fit). Defaults to CPU on purpose — for a single 80x80 frame, GPU transfer overhead exceeds the compute saved.

Training

20,000 synthetic frames: the clean ship rendered at a random heading with real game occluders composited on top. Labels come free by rotating the canonical part map, so nothing was hand-annotated — and only visible parts are labelled, teaching the network to segment what it can actually see. Selected by lowest heading error on the real frames, not by pixel accuracy. 473k parameters, 22 epochs.

Honest limitations

• The 9 real occluded frames are the only real labelled data, and their backgrounds and the portrait asset were harvested to build the training occluders. Expect some optimism; the true test is fresh frames.
• Fixed to one ship sprite at one scale. A different icon needs retraining and a new canonical map.
• The green-based masks (ship_green, open_mask_from_crop) are tuned to this game's palette and will not transfer unchanged.

A note on the labels

One dataset frame (t083) was originally labelled 210 deg, which is physically impossible — it places the ship on open water. This model read 174 deg; a heading-regression CNN and a template matcher both answered ~211 deg, i.e. they made the same mistake as the bad label. The label was corrected to 174. A black-box model agreeing with a label is not evidence the label is right.

License and provenance

Code and weights are Apache-2.0. The canonical ship sprite in canonical.npz and the validation frames are derived from a commercial game's artwork; those underlying assets are not covered by this license and remain the property of their rights holder. Published for research and educational use.

Source: https://github.com/fortunatetumbleweed-commits/ship_direction