BitPixelLM

BitPixelLM is a small autoregressive language model trained to generate 32×32 pixel art from short text prompts. It was built entirely from scratch as a research and learning project, with no pre-trained weights or external image encoders.

Example prompts: a red pixel art sword · a blue pixel art knight · a green pixel art dragon

---

Model Architecture

BitPixelLM is a text-conditioned autoregressive decoder based on BitNet b1.58. Instead of full-precision weights, the decoder uses ternary weights {−1, 0, +1}, making it extremely parameter-efficient.

Key design choices:

• BitLinear b1.58: decoder attention and feed-forward weights are quantized to {−1, 0, +1} during the forward pass via sign(W / mean(|W|)). Embeddings, norms, and the text encoder remain FP32.
• RMSNorm instead of LayerNorm (pre-norm architecture).
• SwiGLU activation in feed-forward blocks.
• 2D positional encoding: the 32×32 pixel grid is encoded with separate row/column sinusoidal embeddings, injected at each decoder layer.
• Cross-attention: the decoder attends to text encoder outputs at every layer.
• Palette tokenization: each pixel is one discrete token from a learned 256-color palette. Generation is autoregressive over 1,024 pixel tokens (32×32) per image.

---

Training

The model was trained on a fully synthetic procedurally-generated dataset of labeled 32×32 pixel art sprites.

Categories include characters, creatures, items, terrain, food, vehicles, buildings, nature, UI icons, furniture, and instruments. Each image has a short descriptive label (e.g. a red pixel art sword) used as the text conditioning signal.

Training configuration:

Results (v3 dataset, best at epoch 32):

---

Usage

Requirements

torch
numpy
Pillow

Load and generate

import json, torch
from PIL import Image
from model.tokenizer import PaletteTokenizer
from model.text_encoder import TextTokenizer, TextEncoder
from model.bit_pixel_decoder import BitPixelLMDecoder, BitPixelLM

# Load tokenizers
palette_tok = PaletteTokenizer(palette_path="palette_256.npy")
with open("vocab.json") as f:
    vocab = json.load(f)
text_tok = TextTokenizer(vocab)

# Build model
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
text_encoder = TextEncoder(
    vocab_size=text_tok.vocab_size, d_model=256, nhead=8,
    num_layers=3, dim_feedforward=512, max_seq_len=32,
)
pixel_decoder = BitPixelLMDecoder(
    vocab_size=palette_tok.vocab_size, d_model=256, nhead=8,
    num_layers=6, dim_feedforward=512, img_size=32,
)
model = BitPixelLM(text_encoder, pixel_decoder).to(device)

# Load weights
ckpt = torch.load("best.pt", map_location=device, weights_only=False)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

# Generate
prompt = "a red pixel art sword"
text_tokens = text_tok.encode(prompt).unsqueeze(0).to(device)
with torch.no_grad():
    generated = model.generate(
        text_tokens,
        sos_token=palette_tok.sos_token,
        eos_token=palette_tok.eos_token,
        temperature=0.8,
        top_k=40,
        top_p=0.9,
    )

# Decode to image
img_array = palette_tok.decode_tokens(generated[0].cpu().tolist())
img = Image.fromarray(img_array, "RGB").resize((256, 256), Image.NEAREST)
img.save("output.png")

Vocabulary

The model knows 222 words covering colors, object types, and descriptors. Prompts that follow the training label pattern (a <color> pixel art <object>) work best. Words outside the vocabulary are silently mapped to <unk>.

Sample supported words: red, blue, green, yellow, orange, purple, gold, dark, teal, silver · sword, shield, bow, axe, staff, wand, armour · knight, wizard, archer, dragon, goblin, skeleton, ghost, vampire · castle, tree, flower, mushroom, chest, potion, gem, key, crown, ship, horse, and more.

---

Limitations

• Outputs are 32×32 pixels. Upscale with Image.NEAREST to preserve the pixel art look.
• Trained entirely on procedurally-generated synthetic data — no real-world artwork knowledge.
• Generation quality is best for prompts close to training label patterns.
• Color fidelity is bounded by the 256-color learned palette.

---

Citations

@article{wang2023bitnet,
  title={BitNet: Scaling 1-bit Transformers for Large Language Models},
  author={Wang, Hongyu and others},
  journal={arXiv:2310.11453},
  year={2023}
}

@article{ma2024bitnet158,
  title={The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits},
  author={Ma, Shuming and others},
  journal={arXiv:2402.17764},
  year={2024}
}

---

License

MIT