加载中…
加载中…
A multi-modal image quality assessment system that combines BRISQUE, Aesthetic Predictor, and CLIP features to predict human-like quality judgments on a 1-10 scale.
This model fuses three complementary approaches to comprehensive image quality assessment:
The fusion network learns optimal weights to combine these diverse quality signals, producing human-like quality judgments that correlate strongly with subjective assessments.
pip install torch torchvision huggingface_hub opencv-python pillow open-clip-torch
# Define a minimal loader class that matches the uploaded head (512 -> 256 -> 1)
import torch
import torch.nn as nn
from huggingface_hub import PyTorchModelHubMixin
class IQFModel(nn.Module, PyTorchModelHubMixin):
def __init__(self, in_dim=512, hidden=256, **kwargs):
# Accept either in_dim/hidden or clip_embed_dim/hidden_dim from config.json
in_dim = kwargs.pop("clip_embed_dim", in_dim)
hidden = kwargs.pop("hidden_dim", hidden)
super().__init__()
self.mlp = nn.Sequential(
nn.Linear(in_dim, hidden),
nn.ReLU(),
nn.Linear(hidden, 1),
)
def forward(self, x):
return self.mlp(x)
# Load weights from the Hub (defaults to model.safetensors)
model = IQFModel.from_pretrained("matthewyuan/image-quality-fusion", map_location="cpu")
model.eval()
# Smoke test on a dummy 512-d vector
with torch.no_grad():
y = model(torch.randn(1, 512)).item()
print(f"score: {y}")
import torch
import torch.nn as nn
from PIL import Image
import open_clip
from huggingface_hub import PyTorchModelHubMixin
# Minimal loader class (same as above)
class IQFModel(nn.Module, PyTorchModelHubMixin):
def __init__(self, in_dim=512, hidden=256, **kwargs):
in_dim = kwargs.pop("clip_embed_dim", in_dim)
hidden = kwargs.pop("hidden_dim", hidden)
super().__init__()
self.mlp = nn.Sequential(
nn.Linear(in_dim, hidden),
nn.ReLU(),
nn.Linear(hidden, 1),
)
def forward(self, x):
return self.mlp(x)
# 1) Load CLIP ViT-B/32 image encoder (512-d output)
clip_model, _, clip_preprocess = open_clip.create_model_and_transforms(
"ViT-B-32", pretrained="openai"
)
clip_model.eval()
# 2) Load the fusion head from the Hub
fusion = IQFModel.from_pretrained("matthewyuan/image-quality-fusion", map_location="cpu")
fusion.eval()
def image_to_clip_embedding(img: Image.Image) -> torch.Tensor:
x = clip_preprocess(img).unsqueeze(0) # [1, 3, H, W]
with torch.no_grad():
feat = clip_model.encode_image(x) # [1, 512]
feat = feat / feat.norm(dim=-1, keepdim=True)
return feat
def predict_quality(image_path: str) -> float:
img = Image.open(image_path).convert("RGB")
emb = image_to_clip_embedding(img) # [1, 512]
with torch.no_grad():
score = fusion(emb).item() # scalar
return float(score)
print("score:", predict_quality("test.jpg"))
Evaluated on the SPAQ dataset (11,125 smartphone images with human quality ratings):
| Metric | Value | Description |
|---|---|---|
| Pearson Correlation | 0.520 | Correlation with human judgments |
| R² Score | 0.250 | Coefficient of determination |
| Mean Absolute Error | 1.41 | Average prediction error (1-10 scale) |
| Root Mean Square Error | 1.69 | RMS prediction error |
| Method | Correlation | R² Score | MAE |
|---|---|---|---|
| Fusion Model | 0.520 | 0.250 | 1.41 |
| BRISQUE Only | 0.31 | 0.12 | 2.1 |
| Aesthetic Only | 0.41 | 0.18 | 1.8 |
| CLIP Only | 0.28 | 0.09 | 2.3 |
The fusion approach significantly outperforms individual components.
Input Image (RGB)
├── OpenCV BRISQUE → Technical Quality Score (0-100, normalized)
├── LAION Aesthetic → Aesthetic Score (0-10, normalized)
└── OpenAI CLIP-B32 → Semantic Features (512-dimensional)
↓
Feature Fusion Network
┌─────────────────────────┐
│ BRISQUE: 1D → 64 → 128 │
│ Aesthetic: 1D → 64 → 128│
│ CLIP: 512D → 256 → 128 │
└─────────────────────────┘
↓ (concat)
Deep Fusion Layers (384D → 256D → 128D → 1D)
Dropout (0.3) + ReLU activations
↓
Human-like Quality Score (1.0 - 10.0)
If you use this model in your research, please cite:
@misc{image-quality-fusion-2024,
title={Image Quality Fusion: Multi-Modal Assessment with BRISQUE, Aesthetic, and CLIP Features},
author={Matthew Yuan},
year={2024},
howpublished={\url{https://huggingface.co/matthewyuan/image-quality-fusion}},
note={Trained on SPAQ dataset, deployed via GitHub Actions CI/CD}
}
# Clone repository
git clone https://github.com/mattkyuan/image-quality-fusion.git
cd image-quality-fusion
# Install dependencies
pip install -r requirements.txt
# Run training
python src/image_quality_fusion/training/train_fusion.py \
--image_dir data/images \
--annotations data/annotations.csv \
--prepare_data \
--epochs 50
This model is automatically deployed via GitHub Actions:
This project is licensed under the MIT License - see the LICENSE file for details.
This model was trained and deployed using automated CI/CD pipelines for reproducible ML workflows.