Velvet2-3B-Vision

Introducing Velvet2

Velvet2 is the new generation of state of the art models developed by Almawave Labs, designed to provide a unified platform for multilingual, multimodal, and enterprise AI applications.

Built from scratch and trained on the Leonardo supercomputing infrastructure operated by CINECA, the Velvet2 family introduces substantial advances in language understanding, structured generation, long-context processing, reasoning and multimodal intelligence, while maintaining a strong focus on deployment efficiency and real-world enterprise requirements.

The Velvet2 family is designed around a common technological foundation and will be released in multiple configurations, spanning compact language models, multimodal models, and larger mixture-of-experts architectures. This approach enables organizations to adopt the most suitable model for their latency, cost, and capability requirements while maintaining architectural consistency across deployments.

Depending on model size and target capabilities, Velvet2 models are pretrained on datasets ranging from approximately 5 trillion to over 8 trillion tokens, followed by extensive post-training pipelines focused on instruction following, reasoning, structured outputs, alignment, and safety.

Velvet2-3B-Vision

Velvet2-3B-Vision is a multimodal member of the Velvet family of models developed by Almawave Labs. Built upon the Velvet2-3B language model, it extends the capabilities of the language backbone with image understanding, enabling joint processing of visual and textual inputs and generation of text outputs.

The model combines the multilingual reasoning, instruction-following, and conversational abilities of Velvet2-3B with a dedicated vision encoder and multimodal projection layers.

Unlike many open-source vision-language models that rely primarily on English supervision, Velvet2-3B-Vision was trained on a deliberately multilingual corpus spanning Italian, English, Portuguese, French, German, and Spanish.

The training corpus was evenly balanced between English and Italian, with approximately 50% of the data in each language, with the remaining 50% distributed across Portuguese, French, German, and Spanish.

Velvet2-3B-Vision supports context lengths of up to 32,768 tokens, in continuity with the Velvet2-3B base text model.

Training was performed on the HPC Leonardo infrastructure hosted by CINECA.

Model details

Velvet2-3B-Vision retains the properties and multilingual language capabilities of Velvet2-3B, while extending the model with a vision encoder for multimodal understanding.

Architecture

Velvet2-3B-Vision consists of 3 main components: a vision encoder, a multimodal projector, and a language model. It combines the multilingual language capabilities of Velvet2-3B with a SigLIP2 vision encoder. Visual embeddings produced by the vision encoder are projected into the language model representation space through a trainable two-layer MLP projector, enabling joint reasoning over image and text inputs.

Vision encoder and visual tokenization

The Vision encoder SigLIP2 NaFlex (so400m: 27 transformer layers, hidden size 1,152, 16 attention heads) uses 16×16-pixel patches; the model takes the 1,152-dimensional patch features of the last layer, without pooling. Views preserve their aspect ratios and can contain up to 1,024 patches. Features are pixel-shuffled in 2×2 groups: four neighbouring patch vectors are concatenated into a 4,608-dimensional feature, reducing the visual sequence by a factor of four. A full 1,024-patch view therefore yields 256 visual tokens, each corresponding to a 32×32-pixel area in the view. The vision encoder (vision tower only, 0.43B parameters) was kept frozen in both training stages.

Projector

The trainable projector maps each 4,608-dimensional visual feature to the language model's 2,560-dimensional embedding space.

The projector was trained from scratch during visual-language alignment and further tuned during instruction tuning with a learning rate, 10× lower than that of the LoRA adapters and 50× lower than in alignment.

Language model and LoRA

The tokenizer and multilingual language capabilities are inherited from Velvet2-3B. The decoder has 38 layers, hidden size 2,560, MLP intermediate size 8,960, 40 attention heads with grouped-query attention over 10 key/value heads (head dimension 64, so k_proj/v_proj project to 640), RoPE with θ = 100,000, and an untied output layer.

During multimodal instruction tuning, LoRA was applied to all 38 decoder layers. The base model weights remained frozen.

In the released checkpoint the adapters are merged into the base weights, so no PEFT dependency is needed at inference.

Image preprocessing

The checkpoint's VelvetVLMImageProcessor plans view dimensions and visual-token counts from image dimensions before pixel encoding. Training, evaluation, and inference use the same processor. Images are EXIF-oriented, converted to RGB (transparent regions composited on white), resized with bilinear antialiased interpolation to dimensions divisible by 32, normalized to [-1, 1], and split into 16×16 patches. Views are encoded separately without square cropping or padding.

Native views, tiling, and tokens

The default image budget is max_tiles=6, where one tile area is 1,024 patches (the area of 512×512 pixels). Images that fit are encoded in one aspect-preserving native view. Small images may be enlarged toward the budget, up to 3× per side; slightly oversized images are downscaled to fit. Larger images are resized while preserving aspect ratio, divided into balanced row-major windows, and accompanied by a global thumbnail. When only one tile area is needed, the image uses one native view without a thumbnail.

After pixel shuffle, each view yields at most 256 visual tokens. Each <image> marker is expanded to <|image_bos|>, the image's visual tokens, and <|image_eos|>. With the default budget, the maximum is 1,792 visual tokens per image (6 windows + thumbnail); with max_tiles=12, it is 3,328. For prompts with multiple images, the tile-area budget is shared: each image receives max(1, max_tiles // number_of_images) tile areas, and each windowed image adds its own thumbnail. Per-page resolution therefore drops as pages are added: with the default max_tiles=6, two A4 pages get 4 windows each (2,012 visual tokens in total), while from four pages on each page is a single native view of about 250 tokens. For multi-page documents, raise the budget per call, e.g. vision_processing_overrides={"max_tiles": 24}. During training, samples longer than 8,192 tokens were fitted by lowering the image budget; at inference no such reduction is applied, so long multi-page prompts must fit the serving context length.

When an image is split into windows, the block also carries each view's position as plain text, in the style of Idefics3 and NVLM: every window is preceded by a <row_r_col_c> tag, rows of windows are separated by a newline, and the thumbnail is preceded by \n\n<global-img>:

<|image_bos|><row_1_col_1>[256 tokens]<row_1_col_2>[256 tokens]\n<row_2_col_1>[256 tokens]<row_2_col_2>[256 tokens]\n\n<global-img>[thumbnail tokens]<|image_eos|>

The tags are ordinary text tokens (no new embeddings), about 8 tokens per window; a single native view has no tags. They let the language model know where each window sits in the image, e.g. where a line of text continues across windows. Training, evaluation, and inference build the block identically from the checkpoint's vision_processing.view_tags setting.

The image block column counts <|image_bos|>, <|image_eos|>, the visual tokens and the view tags, i.e. the context actually taken by the image.

max_tiles controls the maximum tile-area budget for image windows; the thumbnail is additional. It bounds the total window area (in units of 1,024 patches), not the number of windows: tall or wide images can use more, slightly smaller windows, as in the infographic example above (26 windows of 912–960 patches, 24,336 patches in total).

zoom (default 1, maximum 8) plans the image as if enlarged by that factor before applying the same budget. Higher zoom helps with small text only while budget remains. The total enlargement of small images is capped at 3× per side, or the zoom factor when greater.

Recommended image settings used for training domains and the corresponding evaluations:

Infographic training data (tall, dense images) used max_tiles=24 with zoom=1.5: about twice as many 512×512 windows as max_tiles=12, i.e. roughly twice the pixel density on the page.

Training data and process

Training used two stages:

1. Multimodal alignment: The projector alone was trained on 905,424 image-text samples per epoch: dense image captions in six languages (605,562), multilingual short captions (149,900), and transcriptions of paragraphs cropped from born-digital PDF documents (150,000). The vision encoder and language model were frozen.
2. Instruction tuning: The projector and LoRA adapters were trained on 1,254,078 multimodal instruction samples per epoch. Data included public datasets, multilingual synthetic data, multi-turn conversations (up to eight questions about an image), multi-image examples, document understanding, OCR, VQA, chart understanding, and captioning.

Training and evaluation used domain-specific image budgets as described above. Evaluation benchmark samples were removed from training data by image matching.

Capabilities

Velvet2-3B-Vision has been optimized for the following capabilities:

• Image Captioning
• Visual Question Answering (VQA)
• Document Understanding
• OCR-Enhanced Reasoning
• Information Extraction
• Visual Reasoning
• Chart, Diagram, and Table Understanding
• Object Counting & Spatial Understanding
• Multi-image understanding & Reasoning

Supported Languages

The languages that Velvet2-3B-Vision supports are:

• Italian, English, German, Spanish, French, Portuguese (Brazilian Portuguese)

To ensure high-quality multilingual performance, the dataset was deliberately curated to balance linguistic representation and reduce the dominance of English commonly observed in vision-language models, providing strong multimodal capabilities across European languages.

Intended Use

Velvet2-3B-Vision is designed as a general-purpose multilingual vision-language model capable of jointly understanding images and text. The model can be integrated into AI applications requiring multimodal reasoning, visual understanding, document analysis, and image captioning. It is important to note that specific applications may need further model adaptations or additional safeguards to prevent undesirable behavior or outputs.

Evaluation

Evaluation used VLMEvalKit with greedy decoding and exact_matching (no LLM-as-judge), using the recommended image budgets above. Scores below are from the model-card evaluation run.

Comparison values for competing models were collected from the public VLMEvalKit / Hugging Face Open VLM Leaderboard and, where leaderboard entries were unavailable, from the corresponding official model cards. All results are reported as published by the original sources.

The comparison focuses on representative European and Italian vision-language models (Velvet, Salamandra, and Villanova), complemented by widely adopted open-source international baselines (PaliGemma, Gemma3, AKI, and Janus-Pro) to provide additional reference points on standard VLMEvalKit benchmarks.

Comparison with European vision-language models

Salamandra-VL-7B and Villanova-2B-VL-2603 are among the most prominent European vision-language models with publicly reported benchmark results. While their language coverage differs, they provide useful reference points for evaluating the capabilities of Velvet2-3B-Vision across multilingual and multimodal tasks.

Velvet2-3B-Vision supports English, Italian, Portuguese, French, German, and Spanish. Villanova-2B-VL-2603's reported evaluation covers English, Italian, French, German, and Spanish, while Salamandra-VL-7B-2512 supports a broader set of languages, including all six languages supported by Velvet2 family.

Velvet2-3B-Vision outperforms both Salamandra-VL-7B and Villanova-2B-VL-2603 on nearly all shared benchmarks while using fewer parameters than Salamandra-VL-7B.

These results position Velvet2-3B-Vision among the strongest open European vision-language models in the 2-4B parameter range, combining competitive benchmark performance, multilingual capabilities, and a compact architecture.

Multilingual image captioning across European languages

Evaluation is conducted on the COCO35L validation set. Results are reported for English (EN), Italian (IT), Portuguese (PT), Spanish (ES), French (FR), German (DE), and the reported average. Portuguese results are unavailable for Villanova.

Across all evaluated languages and captioning metrics, Velvet2-3B-Vision consistently achieves the strongest results among the compared European models. Performance remains stable across English, Italian, Portuguese, Spanish, French, and German, suggesting effective multilingual transfer from the balanced training mixture.

Multilingual VQA performance across Romance languages

The following results compare multilingual visual question answering (VQA) performance across four Romance languages using two established multilingual VQA benchmarks.

CVQA is used for Spanish (ES) and Portuguese (PT), while MTVQA is used for Italian (IT) and French (FR).

ANLS scores measure answer similarity and provide a direct indication of multilingual visual reasoning capability across Latin-language settings.

A clear distinction emerges between the two benchmarks. On CVQA, Velvet2-3B-Vision achieves the highest ANLS scores in both Spanish and Portuguese, demonstrating strong multilingual visual reasoning capabilities. On MTVQA, Jina-VLM attains the best performance in Italian and French, while Velvet maintains competitive results across both languages. Taken together, these findings indicate that Velvet provides one of the most balanced multilingual profiles among models of comparable size, showing robust cross-lingual generalization across Romance-language VQA tasks.

Usage

Velvet2-3B-Vision can be served with:

Transformers The checkpoint includes the model, processor, and image-processor code; load with `trust_remote_code=True`. The quickest way is a pipeline (images as a URL, a local path or a PIL image):

from transformers import pipeline

pipe = pipeline("image-text-to-text", model="AlmawaveLabs/Velvet2-3B-Vision", trust_remote_code=True)
messages = [{"role": "user", "content": [
    {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
    {"type": "text", "text": "Describe the content of this image in detail."},
]}]
print(pipe(text=messages, max_new_tokens=256, return_full_text=False)[0]["generated_text"])

This example uses the processor directly, with the processor-provided chat template and per-call image budget overrides:

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "AlmawaveLabs/Velvet2-3B-Vision"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(model_id, trust_remote_code=True, dtype=torch.bfloat16)
model = model.to("cuda", dtype=torch.bfloat16).eval()  # also casts the projector LayerNorm, which loads in fp32

image = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"
# a URL, a local path or a PIL image; EXIF orientation and transparency are handled by the processor
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Describe the content of this image in detail."}]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

inputs = processor(
    text=[text],
    images=[image],
    return_tensors="pt",
    return_token_type_ids=False,
    # Optional per-call image budget (see "Image preprocessing"), e.g. vision_processing_overrides={"max_tiles": 12} for a
    # document page, or {"max_tiles": 24} for multi-page documents, {"zoom": 2} for charts.
    vision_processing_overrides=None,
).to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
response = processor.decode(output[0], skip_special_tokens=True)
print(f"Velvet2-3B-Vision response: {response}")

Several images can be passed in one prompt by adding one {"type": "image"} entry per image to the message content and the images, in the same order, to images.

The examples were tested with transformers==5.15: the processor one with torch==2.6, the pipeline with torch==2.8 on CUDA and torch==2.14 on Apple GPUs (MPS), where torch==2.6 lacks the attention the model needs.

vLLM vLLM support for Velvet2-3B-Vision requires specific customization to handle image inputs correctly. Ollama Support for GGUF format will be available soon.

Recommended parameters

Vision processing overrides

Responsible AI

Almawave Labs is signatory of the EU GPAI Code of Practice, see https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai

Velvet2-3B incorporates safety measures throughout dataset preparation, training, and post-training stages.

These measures include data filtering, quality controls, alignment-oriented instruction tuning, and safety-focused evaluation procedures.

Despite these efforts, no language model can guarantee complete prevention of harmful, inappropriate, or inaccurate outputs. Developers are responsible for implementing application-specific safeguards and conducting appropriate risk assessments before deploying the model in production environments.

Large language models are versatile technologies designed to serve a wide range of applications. However, they are not intended to meet every developer's safety requirements out-of-the-box, as these requirements naturally vary depending on the specific use case and application context.

Safety

For our instruction-trained model, we have undertaken comprehensive exercises, engaged in adversarial internal and external evaluations, and put into place mitigation techniques to reduce risks. These exercises were designed to thoroughly examine the model's limitations and potential, simulating real and hypothetical scenarios where undesirable behavior might arise.

However, despite these efforts, it is inevitable that some residual hazards are bound to exist, as every large language model presents intrinsic complexities that cannot be completely eliminated.

Then, developers are advised to implement suitable safety measures and exercise due diligence, tailoring these safeguards to align with their product policies and the specific requirements of their applications.

Some trade-offs between model helpfulness and alignment are likely inevitable. Developers should thoughtfully balance the benefits of alignment and helpfulness for their specific applications and audiences. They must also remain aware of residual risks when using Velvet models and leverage additional safety tools as necessary to achieve an appropriate safety standard for their use case. We advise developers to carefully evaluate risks in the context of their specific use case. They should consider the potential implications of a model failure in their applications and put adequate measures in place to manage such eventualities.

In parallel, we are collaborating with the scientific and industrial community to establish AI safety benchmark standards that are transparent, rigorous, and interpretable. The goal is to promote a better understanding of the risks associated with large language models and support the development of safer and more responsible solutions.

Governance and Internal Oversight

Almawave Labs has established an internal governance framework for the management and continuous oversight of the Velvet model family. Key governance elements include:

Supervision by an Ethical and Technical Committee to ensure the model aligns with principles of transparency, fairness, and safety. Ongoing bias monitoring through auditing tools, with iterative updates to improve alignment with ethical guidelines. Restrictions on commercial and institutional usage to ensure compliance with regulatory frameworks and shared responsibility principles. Periodic review processes to assess the model’s impact in high-risk applications.

Bias, Risks and Limitations

Velvet has been trained on a dataset that, despite all the data curation efforts, might include toxic language and societal biases. This means that models in the Velvet family may reproduce these biases and produce harmful responses when prompted with such inputs. This is a common issue in AI models trained on large datasets, as they can inadvertently perpetuate the biases present in the data. Furthermore, the model may generate inaccurate, incomplete, or redundant responses, which could be socially unacceptable or undesirable, even if the input prompt is not explicitly offensive. This is a potential flaw in the model's design and training process, and it underscores the importance of careful validation and monitoring of AI systems to ensure that they are functioning as intended.

Additionally, using the recommended prompt template is crucial to mitigate the risk of harmful responses, as it is designed to guide the model towards more appropriate and safe outputs. However, it is important to note that the model's performance may still vary depending on the specific context and complexity of the input prompt.

Finally, when using this model in an agentic workflow, it is essential to validate that all imported packages and dependencies are from trusted sources to ensure the model's security and integrity. This is a critical step in maintaining the model's ethical and responsible use, and it is important to prioritize end-to-end security measures to prevent any potential vulnerabilities or breaches.

Future versions of Velvet will integrate automated red-teaming protocols, continuously stress-testing the model against adversarial prompts to identify and mitigate emerging risks.

Sensitive Data Handling and Usage Restrictions

The Velvet model has not been trained on unauthorized personal data and must not be used to process sensitive data without appropriate security measures.

Usage Restrictions:

• Prohibited use on sensitive healthcare, financial, or government data without specific safeguards.
• Mandatory human validation in scenarios where the model’s outputs could have legal or ethical consequences.
• High-risk applications (legal, medical, public governance) must implement content filtering and auditing techniques to ensure response quality and safety.

Ethical Considerations

Almawave Labs core values are openness, inclusivity, and helpfulness. We aim to create AI that is accessible and beneficial for everyone, regardless of their background. Velvet models are designed to be respectful of diverse perspectives and avoid unnecessary judgments. Therefore, Velvet models are designed to be inclusive and respectful of diverse perspectives and needs. We strive to avoid unnecessary judgment or the imposition of normative views, recognizing that content deemed problematic in some contexts can have valuable applications in others.

We deeply respect the dignity and autonomy of all users, particularly their right to free thought and expression, which are fundamental to innovation and progress.

While we have taken significant steps to ensure the safety and reliability of Velvet models, it is important to acknowledge that they may occasionally generate inaccurate, biased, or unsafe responses.

Almawave Labs is actively engaging with ethics committees and domain experts to ensure continuous oversight of Velvet’s outputs, improving safeguards through community feedback.

We strongly encourage the community to exercise caution and conduct thorough safety testing and fine-tuning when using Velvet models for specific tasks.

Opinions expressed by Velvet depend on training data and do not reflect any opinions of Almawave Labs.

Contributions

In addition to contributors to Velvet2-3B

Vision team: Marco Bernardi, Lorenzo Braccaioli, Simone Grazioso, Simone Rossetti, Graziano Specchi

Direction: Andrea Favalli, Raniero Romagnoli.