MTG art CNN: colour identity + card type from the artwork alone

A small convolutional network trained from random initialisation in plain PyTorch (no pretrained weights, no torchvision models) that looks at the art crop of a Magic: The Gathering card and predicts two labels with two softmax heads:

• Colour identity (7): White, Blue, Black, Red, Green, Colorless, Multicolor
• Card type (7): Creature, Planeswalker, Land, Artifact, Enchantment, Instant, Sorcery

It is the final project for the Skillspire AI & Machine Learning course (Nathan King). The live demo is the Space BuildITSystems/mtg-art-demo.

Architecture

TwoHeadCNN(width=32, blocks=[2, 2, 2, 2], dropout=0.2): a 5x5 stride-2 stem, four pre-activation residual stages (widths 32/64/128/256, stride 2 each, 8 BasicBlocks), global average pooling and two linear heads; 2,800,366 parameters, 5.6 MB in bf16. Input: the art crop resized to 160x224 (HxW), normalised with the training mean/std stored in config.json. The complete definition is model.py in the demo Space (BuildITSystems/mtg-art-demo), byte-identical to the training code.

Training

36,325 training artworks (one per illustration_id), Scryfall art crops of English paper printings; validation grouped by card, test = whole held-out sets (unseen cards and unseen sets). Loss = 1.0 x L_colour + 1.0 x L_type with ce (label smoothing 0.1); AdamW lr 0.002, weight decay 0.05 (none on BN/bias), 5% linear warmup then cosine to 1%, batch 256, bf16 autocast, grad-norm clip 1.0, early stopping on val mean macro-F1 (patience 6); best epoch ? of 30 planned (val colour macro-F1 nan, type macro-F1 nan). Why these choices: colour classes are balanced but type is heavily skewed (half of all cards are creatures, ~1% planeswalkers), so macro-F1 drives model selection and the loss is the main experiment dial.

Evaluation

Unseen test split (3,401 art crops from held-out sets), checkpoint best:

Per-class F1 (type head): Creature 0.689 (n=1606), Planeswalker 0.217 (n=37), Land 0.597 (n=347), Artifact 0.211 (n=314), Enchantment 0.103 (n=418), Instant 0.116 (n=393), Sorcery 0.067 (n=286)

Per-class F1 (colour head): White 0.296 (n=520), Blue 0.430 (n=517), Black 0.334 (n=521), Red 0.492 (n=511), Green 0.486 (n=517), Colorless 0.194 (n=257), Multicolor 0.183 (n=558)

Single-image latency (100 images, batch 1, resize + normalise + forward + softmax on a decoded image, JPEG decode excluded, timeit.default_timer; instructor benchmark 0.074 s):

How to use

import json, torch
from PIL import Image
from safetensors.torch import load_file
from model import TwoHeadCNN            # model.py from this repo

cfg = json.load(open("config.json"))
model = TwoHeadCNN.from_config(cfg["model"]).eval()
model.load_state_dict(load_file("model.safetensors"))
H, W = cfg["image_size"]
img = Image.open("art_crop.jpg").convert("RGB").resize((W, H), Image.BILINEAR)  # bilinear, as in training/eval
x = torch.tensor(list(img.getdata()), dtype=torch.float32).view(H, W, 3).permute(2, 0, 1) / 255
x = (x - torch.tensor(cfg["normalization"]["mean"]).view(3, 1, 1)) / torch.tensor(cfg["normalization"]["std"]).view(3, 1, 1)
with torch.no_grad():
    color_logits, type_logits = model(x[None])
print(cfg["labels"]["color"][color_logits.argmax().item()], cfg["labels"]["type"][type_logits.argmax().item()])

Limitations

• Trained on official card art only; photos of physical cards, sleeves, glare and perspective are out of distribution (the Space crops a photo with a fixed template, which is approximate).
• The type head reads style cues (a creature portrait vs a landscape), not the type line; lands and sorceries with similar art are confused, planeswalkers are rare.
• Colour is the colour identity of the card the art belongs to, which the art does not always show.

Legal

This project is unofficial Fan Content permitted under the Wizards of the Coast Fan Content Policy. Not approved/endorsed by Wizards. Portions of the materials used are property of Wizards of the Coast. (c) Wizards of the Coast LLC.

Card metadata and art crops were obtained from Scryfall under its API terms; Scryfall is not affiliated with this project. The training image set is not redistributed; the Space ships six test-split art crops as examples under the Fan Content Policy, and only the trained weights are published here.