GRAFT-1B-Vision-GGUF

GRAFT-1B can now see images, with zero vision training.

This repo pairs the GRAFT-1B language model (SysPrompt release, Q6_K) with a SigLIP-so400m image encoder and a small 2-layer connector, packaged as a llama.cpp mmproj file. Nothing was trained to make this work. GRAFT-1B was shrunk from Qwen3-1.7B, and a connector originally trained for Qwen3-1.7B carried over to GRAFT-1B directly.

Made by SmallAICreator / UltraLabs.

Quick start

llama.cpp

# one image, one question
llama-mtmd-cli -m GRAFT-1B-SysPrompt-Q6_K.gguf \
  --mmproj mmproj-GRAFT-1B-siglip-so400m-Q8_0.gguf \
  --image photo.jpg -p "Describe this image." --temp 0.4

# chat UI with image upload at http://localhost:8080
llama-server -m GRAFT-1B-SysPrompt-Q6_K.gguf \
  --mmproj mmproj-GRAFT-1B-siglip-so400m-Q8_0.gguf -c 4096

PocketPal (Android / iOS)

Models → + → Add from Hugging Face → search GRAFT-1B-Vision-GGUF → download the GGUF. PocketPal fetches the projector automatically; turn on Vision on the model card, then attach a photo in chat.

Example outputs

Real, unedited outputs (temperature 0.2, no training of any part):

Image: two cats lying on a pink blanket

GRAFT-1B: The image is a close-up of two cats, one on a pink and white pillowcase, the other curled up on a soft, cozy blanket. The contrast between their fur tones and the surrounding fabric creates a warm, intimate scene.

Image: a living room with a person, chairs, plants and a table

GRAFT-1B: The image is a close-up of a living room, with a focus on the furniture and decor. The person is sitting in a chair or standing by a sofa, looking at the colorful arrangement of plants, flowers, and other decorative items.

On a phone

Running fully on-device on a Pixel 6a in PocketPal (this repo's files, Q8_0 vision file), with a photo of a backyard deck:

User: (photo) What's this image?

GRAFT-1B: The image shows a wooden deck or patio with a wooden plank or beam running across it, with a wooden fence or railing on one side and a wooden plank or beam on the other. The dark gray and browns in the image suggest that it's a naturally occurring wood structure, with the dark wood being the type used for the deck or railing.

7.58 tokens/sec generation, about 46 s to the first token (the vision encoder and the 729 image tokens are processed on the phone's CPU first).

Limitations

• Memory: the model plus vision file need about 2 GB of free RAM. On 6 GB phones, close other apps first.

• Zero-shot connector. The connector was trained for Qwen3-1.7B, not GRAFT-1B. It works well for general scenes, objects, animals and colors, but small details are often missed or guessed (in the examples above it missed a pair of TV remotes and a TV).

• Not for reading text. One 384×384 view per image, so small text, documents and screenshots are not read reliably.

• Counting and fine detail are weak.

• Everything in the GRAFT-1B limitations still applies: it is a 1B model and can invent details confidently.

Credits and licenses

• GRAFT-1B language model: SmallAICreator / UltraLabs, Apache-2.0.
• Image encoder: google/siglip-so400m-patch14-384, Apache-2.0.
• Connector (and the encoder weights used with it): taken unchanged from markendo/llava-extract-qwen3-1.7B, the Extract model from Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models (paper, code, MIT). Thanks to the authors.
• Packaged with llama.cpp's LLaVA encoder converter.