ellamind/propella-1-4b

propella-1 is a family of small multilingual LLMs that annotate text documents across six categories: core content, classification, quality & value, audience & purpose, safety & compliance, and geographic relevance. The annotations can be used to filter, select, and curate LLM training data at scale.

Disclaimer: This is a research project, not an official ellamind product. For production-ready evaluation solutions, check out elluminate.

Highlights

• Annotate 18 properties: Covers well-established dimensions like content quality and educational value, plus underexplored ones like reasoning indicators and time-sensitivity.
• Fast & accurate: Small models (0.6B, 1.7B, 4B) that punch above their weight. Trained in fp8, ready for high-throughput inference.
• Any text, any format: Handles web pages, PDFs, code, math, post-training data and more.
• Highly multilingual: Supports 57 languages.

The propella-1 family of models

Properties

propella-1 models evaluate documents across 18 properties organized into six categories:

Read the property reference for detailed definitions and enum values.

Datasets annotated with propella-1

See openeurollm/propella-annotations.

Input

A text document in any of the 57 supported languages.
The model has a 64k context length but we recommend to truncate documents at 50k characters (see usage).

Output

A JSON object containing annotations. The output strictly conforms to a predefined schema with enumerated values for categorical properties.

Example output

{
  "content_integrity": "complete",
  "content_ratio": "mostly_content",
  "content_length": "moderate",
  "one_sentence_description": "Technical documentation explaining how to define and evaluate structured LLM output schemas using elluminate's Python client.",
  "content_type": [
    "technical_documentation",
    "instructional",
    "source_code"
  ],
  "business_sector": [
    "technology_software"
  ],
  "technical_content": [
    "code_heavy"
  ],
  "information_density": "dense",
  "content_quality": "excellent",
  "audience_level": "advanced",
  "commercial_bias": "minimal",
  "time_sensitivity": "slowly_changing",
  "content_safety": "safe",
  "educational_value": "high",
  "reasoning_indicators": "explanatory",
  "pii_presence": "no_pii",
  "regional_relevance": [
    "global"
  ],
  "country_relevance": [
    "none"
  ]
}

Usage

See propella.py for prompts and schemas. We recommend enforcing a strict json schema without any whitespace for error-free generation.

Serving

We recommend serving propella models with SGLang and the llguidance structured output backend:

python -m sglang.launch_server \
    --model-path outputs/propella-1-4b \
    --host 0.0.0.0 \
    --port 8000 \
    --context-length 65536 \
    --max-running-requests 256 \
    --chunked-prefill-size 8192 \
    --enable-mixed-chunk \
    --num-continuous-decode-steps 8 \
    --grammar-backend llguidance \
    --mem-fraction-static 0.7

fp8 on H100

python -m sglang.launch_server \
    --model-path outputs/propella-1-4b \
    --quantization w8a8_fp8 \
    --kv-cache-dtype fp8_e4m3 \
    --host 0.0.0.0 \
    --port 8000 \
    --context-length 65536 \
    --max-running-requests 256 \
    --chunked-prefill-size 8192 \
    --enable-mixed-chunk \
    --num-continuous-decode-steps 8 \
    --grammar-backend llguidance \
    --mem-fraction-static 0.7

For single-node multi-GPU we recommend increasing data-parallel-size. For large scale offline inference on SLURM clusters we use inference-hive.

Sending request via OpenAI SDK

from openai import OpenAI
from propella import (
    create_messages,
    AnnotationResponse,
    get_annotation_response_schema,
)

document = "Hi, its me Max."

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="ellamind/propella-1-4b",
    messages=create_messages(document),
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "AnnotationResponse",
            "schema": get_annotation_response_schema(flatten=True, compact_whitespace=True),
            "strict": True,
        }
    },
)
response_content = response.choices[0].message.content
result = AnnotationResponse.model_validate_json(response_content)
print(result.model_dump_json(indent=4))

Result

{
    "content_integrity": "complete",
    "content_ratio": "complete_content",
    "content_length": "minimal",
    "one_sentence_description": "A short personal greeting introducing someone named Max.",
    "content_type": [
        "conversational"
    ],
    "business_sector": [
        "general_interest"
    ],
    "technical_content": [
        "non_technical"
    ],
    "information_density": "dense",
    "content_quality": "good",
    "audience_level": "general",
    "commercial_bias": "none",
    "time_sensitivity": "evergreen",
    "content_safety": "safe",
    "educational_value": "none",
    "reasoning_indicators": "none",
    "pii_presence": "contains_pii",
    "regional_relevance": [
        "culturally_neutral"
    ],
    "country_relevance": [
        "none"
    ]
}

Throughput

The throughput results below provide a rough estimate for GPU-hours required to annotate 1M documents. After a short warmup, we run inference for 5k documents, sending 1k concurrent requests to the SGLang server.

Evaluation

We evaluate the propella-1 models on a test set containing 3k documents. For these documents we obtain annotations from Gemini-3-Pro (reasoning_effort: high), which we consider as groundtruth labels under the assumption that they represent the upper limit in terms of annotation quality.

All baseline models use the detailed annotator system- and user-prompts as defined in propella.py. For throughput reasons, the propella-1 models use a very short, propella-1 specific prompt. We also tested some baseline models with the propella-1 prompt, always leading to worse performance as the prompt lacks details.

Metrics by Property Type

Properties are grouped into three categories, each evaluated with an appropriate metric:

• Ordinal Properties (11 properties): QWK (Quadratic Weighted Kappa), which measures agreement while accounting for the ordinal nature of labels. It penalizes larger disagreements more heavily.
• Binary Properties (1 property): F1, the harmonic mean of precision and recall.
• Multi-select Properties (5 properties): IoU (Jaccard Index), intersection-over-union averaged across samples.
• Free-text Properties (1 property): The one_sentence_description property is excluded from quantitative evaluation.

_{Click to view at full size}

Overall Score

The overall score is a weighted average of the primary metric for each property type:

overall = (11/17 × avg_QWK) + (1/17 × avg_F1) + (5/17 × avg_IoU)

Performance with fp8

propella-1 models are trained with fp8 precision and work well in both bf16 and fp8 inference modes. The plot below compares annotation quality across precisions for the 4b and 1.7b models. For the 0.6b model we recommend bf16 precision.

Language Support

The training data for propella-1 contains documents in 57 languages:

Citation

@misc{idahl2026propella1multipropertydocumentannotation,
      title={propella-1: Multi-Property Document Annotation for LLM Data Curation at Scale}, 
      author={Maximilian Idahl and Benedikt Droste and Björn Plüster and Jan Philipp Harries},
      year={2026},
      eprint={2602.12414},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.12414}, 
}

Acknowledgements

• This project is supported by the OpenEuroLLM project, co-funded by the Digital Europe Programme under GA no. 101195233. For more information see openeurollm.eu.
• This project is supported by the LLMs4EU project, co-funded by the Digital Europe Programme under GA no. 101198470. For more information see LLMs4EU website.
• This project is supported by the German Federal Ministry for Economic Affairs and Energy (BMWE) under the soofi (Sovereign Open Source Foundation Models for European Intelligence) project.
• We acknowledge the EuroHPC Joint Undertaking for supporting this project through access to the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium, through an EuroHPC AI Factory Large Scale Access call (EHPC-AIF-2025LS01-028).
• We thank the AI Service Center for Sensitive and Critical Infrastructures (KISSKI), hosted by GWDG, for additional compute access.