NeoMME (800M): Single-Tower Multimodal-Native Multilingual Foundation Encoder

Model summary

NeoMME is an efficient Multilingual and Multimodal-native foundational Encoder. Text tokens and raw image patches pass through one shared Transformer encoder. NeoMME does not use a separately pretrained vision encoder or a causal language model.

NeoMME-800M is a pretrained encoder backbone and cannot be used on its own for a downstream task. It returns contextual token representations, so users should fine-tune a task-specific head for retrieval, classification, extraction, or another downstream task. For document retrieval, use NeoMME-800M-Retriever.

Usage

Use NeoMME with transformers:

# accelerate is an optional dependency needed only when using device_map="auto".
pip install -U accelerate transformers

The example below generates hidden states for a text document and a document image in one forward pass. The hidden states are not usable as is for a downstream task. If you are looking for retrieval embeddings, you should use the NeoMME-800M-Retriever model instead.

Generate hidden states

import requests
import torch
from PIL import Image

from transformers import AutoModel, AutoProcessor


def encode_document_text(processor, text: str) -> str:
    return f"{processor.tokenizer.document_token}{text}"


model_id = "Hcompany/NeoMME-800M"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, device_map="auto")

text = "The cat sat on a mat."
image_url = "https://github.com/tonywu71/colpali-cookbooks/blob/main/examples/data/shift_kazakhstan.jpg?raw=true"
image = Image.open(requests.get(image_url, stream=True).raw)

inputs = processor(
    text=[
        encode_document_text(processor, text),
        encode_document_text(processor, processor.image_token),
    ],
    images=[image],
    padding=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model(**inputs)

text_hidden_states, image_hidden_states = outputs.last_hidden_state

NeoMME was pretrained with a masked discrete-diffusion objective. Therefore, the model can restore masked text tokens given the surrounding text (and, when present, image patches). The example below fills a single mask as a sanity check of that objective. It is not a generative or conversational model.

Masked language modeling

import torch

from transformers import AutoModelForMaskedLM, AutoProcessor


model_id = "Hcompany/NeoMME-800M"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id, device_map="auto")

# Equivalent: "<doc>The capital of <mask> is London."
text = f"{processor.tokenizer.document_token}The capital of {processor.tokenizer.mask_token} is London."
inputs = processor(text=[text], return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model(**inputs)

masked_index = (inputs.input_ids[0] == processor.tokenizer.mask_token_id).nonzero().item()
predicted_token_id = outputs.logits[0, masked_index].argmax(dim=-1)
print(processor.tokenizer.decode(predicted_token_id))

Training

NeoMME-800M was pretrained from scratch on multilingual text and visual-text data, including web text, code, math, document pages, captions, and synthetic OCR data. The model learns to restore masked text tokens. For document images paired with transcripts, image patches remain visible and the pretraining objective has no pixel reconstruction loss.

The NeoMME technical report describes the full pretraining recipe.

Limitations

• NeoMME-800M is a pretrained encoder backbone and requires task-specific fine-tuning.
• The model has not received a comprehensive safety, bias, or privacy evaluation.

License

Model weights are released under the Apache 2.0 license.

Citation

@misc{lac2026neommesingletowermultimodalnativemultilingual,
      title={NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference},
      author={Aurélien Lac and Tony Wu},
      year={2026},
      eprint={2609.01657},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2609.01657},
}