Madrimed1.2 2B GGUF

Optimized GGUF Quantization Tiers for MadriMed 1.2 (2B Medical Mulitmodel)

---

Overview

This repository provides official GGUF-quantized binaries of MadriMed 1.2 (Madrimed1.2-VL-2B), a compact 2B parameter medical vision-language model. These files are optimized for high-performance, low-latency local execution using llama.cpp, llama-server, llama-cpp-python, and Ollama.

---

Model Architecture & Dual-File Requirement

GGUF multimodal pipelines require a dual-file architecture:

1. The Language Backbone (medmax-2b-[quant].gguf): Autoregressive transformer decoder (28 layers) handling clinical text generation and reasoning.
2. The Multimodal Projector (mmproj-[quant].gguf): Vision encoder and DeepStack patch merger (qwen3vl_merger) processing image inputs at a 768 resolution.

Crucial Note: To perform visual inference, you must pass both the quantized model file (-m) and the vision projector file (--mmproj).

---

Complete Quantization Inventory

---

Deployment & Usage Guide

1. Local CLI Inference (llama-cli)

Run quick diagnostic queries directly in your terminal:

!./llama.cpp/build/bin/llama-cli \
    -m medmax-2b-Q6_K.gguf \
    --mmproj mmproj-Q8_0.gguf \
    --image /path/to/scan.jpg \
    -p "Question: What abnormality is visible in this scan? Answer:" \
    --image-min-tokens 1024 \
    -ngl 99

Important Flag: Always include --image-min-tokens 1024 for clinical grounding tasks. Qwen-VL architectures require adequate token allocation to maintain high-resolution perception on subtle lesions.

2. OpenAI-Compatible API Server (llama-server)

Host a local server on your machine for integration into applications:

./llama.cpp/build/bin/llama-server \
    -m medmax-2b-Q6_K.gguf \
    --mmproj mmproj-Q8_0.gguf \
    -c 8192 \
    -np 1 \
    -ngl 99 \
    --port 8080 \
    --image-min-tokens 1024

3. OpenAI SDK

import base64
from openai import OpenAI

client = OpenAI(base_url="[http://127.0.0.1:8080/v1](http://127.0.0.1:8080/v1)", api_key="none")

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_b64 = encode_image("chest_xray.jpg")

response = client.chat.completions.create(
    model="madrimed1.2-2b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Is there evidence of consolidation?"},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}}
            ]
        }
    ],
    max_tokens=128,
    temperature=0.0
)

print(response.choices[0].message.content)

Limitations & Safety Guidelines

Research & Educational Use Only

MadriMed 1.2-GGUF is an experimental research model. It is not certified for autonomous clinical diagnostic decision-making. All model-generated interpretations and visual inferences should be reviewed and independently verified by a licensed medical professional before use in any clinical workflow.