SkyVLM

A 99.4M-parameter vision-language model for satellite imagery, trained from scratch on a single 8 GB laptop GPU. It writes a structured caption such as a satellite image of <main object>, surrounded by <object>; <object>.

Code, training scripts, a Streamlit demo and full results: https://github.com/thorOdinson16/SkyVLM

Model

image ─► ViT (25.7M; masked-image-modeling pretrained, then CLIP-aligned)
          └─ 196 patch tokens ─► 2-layer MLP projector (1.05M) ─► LM (72.6M decoder) ─► caption

The language model was pretrained on ~1B tokens (90% FineWeb-Edu, 10% SkyScript captions), the encoder on 200k unlabelled SkyScript images and then aligned to captions with a CLIP-style loss, and the whole model was fine-tuned on 200k SkyScript image-caption pairs (projector first, then everything).

Usage

This is a custom PyTorch architecture, not a transformers model. Download the repo and run:

pip install torch torchvision safetensors sentencepiece pandas pillow
python inference.py my_image1.jpg my_image2.png

or from Python:

from PIL import Image
from inference import load, caption

model, tok, cfg = load(".")                      # repo directory
print(caption(model, tok, cfg, [Image.open("my_image.jpg")])[0])

Images are resized to 224×224 and decoded greedily. The output follows the SkyScript caption template (OpenStreetMap-style tags).

Results

Official SkyScript test captions, all 30k images, greedy decoding:

Shuffled or blank images drop CIDEr-D to 0.09 / 0.04, so the model uses the image.

Image-text retrieval on 1,000 test pairs, ranking captions by log p(caption | image) (image→text uses a PMI correction):

SkyCLIP is a general remote-sensing model while SkyVLM is a specialist trained on these templated captions, so this is a reference point rather than a claim of a better model.

Vision encoder

RESISC45 linear probe (45 scene classes, 6,300 test images, frozen features; no RESISC45 data seen in training):

Language model

FineWeb-Edu validation perplexity 21.7 (72.6M parameters, 983M training tokens). SkyScript caption validation loss bottomed at 0.76 and ended at 0.94 as the model began to memorize the captions (10% of each batch, ~18 passes).

Sanity checks

Main-object category (building, road, railway, ...) is right 61.5% of the time versus 21% for always guessing the most common one (measured on the variant without CLIP alignment); all outputs follow the caption template.

Single training run per condition; no variance estimates.

Limitations

• Trained only on SkyScript's templated, OpenStreetMap-derived captions: it will not follow free-form prompts or answer questions.
• Typical errors are the right category with the wrong specifics (an apartment block described as an office). Exact main-object match is 21%, and references have several valid descriptions per image, so n-gram and exact-match scores understate quality.
• Images are ~200 px at 224×224 input; fine detail is limited.
• Not evaluated outside SkyScript imagery, on other regions' sensors, or for any safety-critical use.

Data and license

Apache-2.0. Trained on SkyScript (MIT), which pairs Google Earth Engine imagery with OpenStreetMap-derived captions. OpenStreetMap data is © OpenStreetMap contributors (ODbL); imagery remains subject to its providers' terms. FineWeb-Edu (ODC-By) was used for language-model pretraining.