CLIP-KO: Knocking Out Typographic Attacks in CLIP 💪🤖

Finally, a CLIP without a 'text obsession'! 🤗

Love ❤️ this CLIP?

ᐅ Buy me a coffee on Ko-Fi ☕

Or click here for address to send 🪙₿ BTC

3PscBrWYvrutXedLmvpcnQbE12Py8qLqMK

---

🔥 CLIP-KO is very robust against typographis attacks, but may produce OOD Text Embeddings.

• 📝 Read the paper (PDF) here.
• 🌱 If you're looking for a a Text Encoder, you'll probably want the KO-LITE variant instead.
• Still want this one? But be warned of OOD embeddings unless you use CLEAN ENGLISH WORDS (see image examples on the bottom). This model is a nitpicker, and potentially (!) SFW (but I can't gurantee it, please see paper for details)
• ⚠️ Download the STRICT TEXT ENCODER
• 🤓 Wanna fine-tune yourself? Get the code on my GitHub.
• Included: Code for fine-tuning and all benchmarks / claims (as per the paper)

👉 Check out the KO-LITE variant of this model (less strict, but good robustness!)

---

👉 CLICK ME to expand example benchmark code ⚡💻

from datasets import load_dataset
from transformers import CLIPModel, CLIPProcessor
import torch
from PIL import Image
from tqdm import tqdm
import pandas as pd

device = "cuda" if torch.cuda.is_available() else "cpu"

# BLISS / SCAM Typographic Attack Dataset
# https://huggingface.co/datasets/BLISS-e-V/SCAM
ds = load_dataset("BLISS-e-V/SCAM", split="train")

# Benchmark pre-trained model against my fine-tune
model_variants = [
    ("OpenAI ", "openai/clip-vit-large-patch14", "openai/clip-vit-large-patch14"),
    ("KO-CLIP", "zer0int/CLIP-KO-TypoAttack-Attn-Dropout-ViT-L-14", "zer0int/CLIP-KO-TypoAttack-Attn-Dropout-ViT-L-14"),
]

models = {}
for name, model_path, processor_path in model_variants:
    model = CLIPModel.from_pretrained(model_path).to(device).float()
    processor = CLIPProcessor.from_pretrained(processor_path)
    models[name] = (model, processor)

for variant in ["NoSCAM", "SCAM", "SynthSCAM"]:
    print(f"\n=== Evaluating var.: {variant} ===")
    idxs = [i for i, v in enumerate(ds['id']) if v.startswith(variant)]
    if not idxs:
        print(f"  No samples for {variant}")
        continue
    subset = [ds[i] for i in idxs]

    for model_name, (model, processor) in models.items():
        results = []
        for entry in tqdm(subset, desc=f"{model_name}", ncols=30, bar_format="{l_bar}{bar}| {n_fmt}/{total_fmt} |"):
            img = entry['image']
            object_label = entry['object_label']
            attack_word = entry['attack_word']

            texts = [f"a photo of a {object_label}", f"a photo of a {attack_word}"]
            inputs = processor(
                text=texts,
                images=img,
                return_tensors="pt",
                padding=True
            )
            for k in inputs:
                if isinstance(inputs[k], torch.Tensor):
                    inputs[k] = inputs[k].to(device)

            with torch.no_grad():
                outputs = model(**inputs)
                image_features = outputs.image_embeds
                text_features = outputs.text_embeds

                logits = image_features @ text_features.T
                probs = logits.softmax(dim=-1).cpu().numpy().flatten()
                pred_idx = probs.argmax()
                pred_label = [object_label, attack_word][pred_idx]
                is_correct = (pred_label == object_label)

            results.append({
                "id": entry['id'],
                "object_label": object_label,
                "attack_word": attack_word,
                "pred_label": pred_label,
                "is_correct": is_correct,
                "type": entry['type'],
                "model": model_name
            })

        n_total = len(results)
        n_correct = sum(r['is_correct'] for r in results)
        acc = n_correct / n_total if n_total else float('nan')
        print(f"| > > > > Zero-shot accuracy for {variant}, {model_name}: {n_correct}/{n_total} = {acc:.4f}")

---

Typographic Attack / adversarial robustness:

Attention Heatmaps without artifacts:

---

🍂 ALL: Flux.1-dev, NO T5 - CLIP only! CFG=5, Heun, fixed seed. Prompts, in order:

1. "bumblewordoooooooo bumblefeelmbles blbeinbumbleghue" (weird CLIP words / text obsession / prompt injection)
2. "a photo of a disintegrimpressionism rag hermit" (one weird CLIP word only)
3. "a photo of a breakfast table with a highly detailed iridescent mandelbrot sitting on a plate that says 'maths for life!'" (note: "mandelbrot" literally means "almond bread" in German)
4. "mathematflake tessswirl psychedsphere zanziflake aluminmathematdeeply mathematzanzirender methylmathematrender detailed mandelmicroscopy mathematfluctucarved iridescent mandelsurface mandeltrippy mandelhallucinpossessed pbr" (Complete CLIP gibberish math rant)
5. "spiderman in the moshpit, berlin fashion, wearing punk clothing, they are fighting very angry" (CLIP Interrogator / BLIP)
6. "epstein mattypixelart crying epilepsy pixelart dannypixelart mattyteeth trippy talladepixelart retarphotomedit hallucincollage gopro destroyed mathematzanzirender mathematgopro" (CLIP rant)

Evaluation Results