Visual-Qwen — TikTok Sludge Detector
A trimodal sludge-content classifier for short-form videos: BLIP-2 (EVA-CLIP-G/14 + Q-Former) + a Linear projector (frozen) + Qwen3-4B with LoRA, augmented with Whisper V3 Turbo audio transcripts. Identifies "sludge" — short videos that stack multiple unrelated visual feeds on one screen, often paired with mismatched audio, engineered to bypass moderation by exploiting unimodal classifiers.
Performance
Held-out 300-video Kaggle stratified test split (video-level majority vote across 4 sampled frames):
| Metric | Value | 95% CI |
|---|---|---|
| Accuracy | 96.67% | 94.33-98.67 |
| Precision | 95.58% | 92.61-98.31 |
| Recall | 98.86% | 97.06-100.00 |
| F1-score | 97.19% | - |
Confusion matrix (TN=117, FP=8, FN=2, TP=173). 95% CIs from 1,000-iteration bootstrap resampling.
Additional out-of-distribution evaluation on 262 fresh TikToks (scraped May 2026, labelled by Gemma-4-31B-IT): 92.37% accuracy (95% CI 89.31-95.42). See the dataset ood-data/ folder for the raw eval set.
Architecture
input frame ─► EVA-CLIP-G/14 vision encoder (frozen, 1B params)
│
▼
Q-Former + 32 query tokens (frozen)
│
▼
Linear projector (frozen, 4M params) ◄── audio transcript (Whisper V3 Turbo)
│ │
└───────────┬───────────────────┘
▼
Qwen3-4B (4B params) + LoRA r=16 (~32M trainable)
│
▼
{sludge | non-sludge} + explanation
The Linear projector is trained in a stage-1 image-caption pre-training pass and then frozen during LoRA fine-tuning on the sludge corpus. Freezing the projector outperformed training it by +0.77 pp on the same eval — see paper section 4.
Usage
import torch
from transformers import Blip2ForConditionalGeneration, Blip2Processor
MODEL_ID = "alpharomercoma/vqwen-qformer-tiktok-v2"
model = Blip2ForConditionalGeneration.from_pretrained(
MODEL_ID, dtype=torch.bfloat16, device_map="auto",
)
processor = Blip2Processor.from_pretrained(MODEL_ID)
# Single frame + ASR transcript -> sludge / non-sludge
from PIL import Image
frame = Image.open("frame.jpg").convert("RGB")
transcript = "subway surfers gameplay over family guy clip" # from Whisper V3 Turbo
messages = [{"role": "user", "content": [
{"type": "text", "text": f"Audio transcript: {transcript}"},
{"type": "image"},
{"type": "text", "text": "Is this sludge content? Answer yes or no."},
]}]
prompt = processor.tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=prompt, images=frame, return_tensors="pt").to("cuda")
inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Run video-level by sampling 4 frames uniformly across the clip and majority-voting the predictions.
Training data
- 2,000 TikTok and YouTube Shorts videos (1,163 sludge / 837 non-sludge after correction).
- Hand-labelled, then label-refined via a Qwen3-VL-30B-A3B teacher + Gemma-3-27B-IT cross-judge audit pass (231 video flips, 11.55% noise rate in the original GT).
- Multi-task supervision: 25% classify, 15% layout, 20% describe, 35% coupled (classify+explain), 5% honest-refusal templates.
Dataset available on Kaggle — corrected labels, live OOD eval set, and raw mp4s in one dataset.
Code & paper
- Code: github.com/alpharomercoma/vqwen-qformer
- Project page: alpharomercoma.com/thesis
- Live demo: Hugging Face Space
Citation
@inproceedings{coma2026visualqwen,
title={Visual-Qwen: Augmenting Multimodal Deep Learning with Attention Mechanisms for Sludge Content Detection},
author={Coma, Alpha Romer and others},
year={2026},
organization={FEU Institute of Technology}
}