Introducing Velvet2

Velvet2 is the new generation of state-of-the-art models developed by Almawave Labs, designed to provide a unified platform for multilingual, multimodal, and enterprise AI applications.

Built from scratch and trained on the Leonardo supercomputing infrastructure operated by CINECA, the Velvet2 family introduces substantial advances in language understanding, structured generation, long-context processing, reasoning and multimodal intelligence, while maintaining a strong focus on deployment efficiency and real-world enterprise requirements.

The Velvet2 family is designed around a common technological foundation and will be released in multiple configurations, spanning compact language models, multimodal models, and larger mixture-of-experts architectures. This approach enables organizations to adopt the most suitable model for their latency, cost, and capability requirements while maintaining architectural consistency across deployments.

Depending on model size and target capabilities, Velvet2 models are pretrained on datasets ranging from approximately 5 trillion to over 8 trillion tokens, followed by extensive post-training pipelines focused on instruction following, reasoning, structured outputs, alignment, and safety.

Velvet2-3B-Speech

Velvet2-3B-Speech is a speech language model from the Velvet family of large language models, developed by Almawave Labs. Supporting six languages (English, Italian, German, Spanish, French, and Portuguese), it is designed for a wide range of tasks, including automatic speech recognition, speech and spoken-query question answering, speech summarization, speech translation, speech emotion recognition, and function calling.

Velvet2-3B-Speech was trained on the HPC Leonardo infrastructure hosted by CINECA.

The model was trained on nearly 5 trillion text tokens and 100,000 hours of speech data, comprising almost 30 million audio clips.

Model details

Architecture

Velvet2-3B-Speech consists of 3 main components: an audio encoder, an audio projector and a language model.

The audio encoder, Whisper Large v3 encoder, processes spectral features extracted from a raw audio waveform by an audio processor and transforms them into high-level embeddings that capture speech content and relevant acoustic information for downstream processing.

The audio projector performs modality alignment, mapping the acoustic features extracted by the audio encoder onto the embedding space of the large language model. The audio projector was trained from scratch.

License

Velvet2-3B-Speech is made available under the Apache 2.0 license

Supported Languages

The languages that Velvet2-3B-Speech supports are:

• English, Italian, German, Spanish, French, Portuguese

To ensure high-quality multilingual performance, the dataset was curated to balance linguistic representation, reducing overfitting biases.

Intended Use

Velvet2-3B-Speech is designed to be integrated into AI systems or applications. It is important to note that specific applications may need further model adaptations or additional safeguards to prevent undesirable behavior or outputs.

Capabilities

• Audio (speech) mode
  • Automatic Speech Recognition (ASR)
  • Spoken Query Question Answering (SQQA)
  • Speech Question Answering (SQA)
  • Speech Summarization (SS)
  • Speech Translation (ST)
  • Speech Emotion Recognition (SER)
  • Function calling (FC)
• Text only mode. Velvet2-3B-Speech was specifically designed to preserve the capabilities of Velvet2-3B on text-only inputs. During inference, the LoRA adapters for the Velvet2-3B-Speech language model activate only when the input includes audio, thus preserving the behavior of Velvet2-3B when input is text-only.

Training

The data used to train the audio projector and fine-tune the language model was selected from nearly 30 million audio examples in English, Italian, German, Spanish, French, and Portuguese, for a total duration of approximately 95,000 hours. The full training set combines open datasets datasets containing LLM-generated responses and datasets containing TTS-generated audio.

Training consists of two stages: pre-training and fine-tuning.

Pre-training

The goal of the pre-training stage is to train the audio projector from scratch. The weights of all other components were kept frozen.

For this stage, ASR, ST, and a subset of SQQA examples shorter than 2 minutes were used. The three tasks were equally represented, while English and Italian (each accounting for approximately 20% of the data) were intentionally sampled more heavily than the other four languages, each accounting for approximately 15% of the training data.

Fine-tuning

During the fine-tuning stage, both the audio projector and the LLM were adapted by training LoRA adapters.

The audio projector was further fine-tuned to better accommodate longer audio inputs.

The fine-tuning data include:

• A subset of the most carefully curated ASR, ST, and SQQA data used during pre-training. This subset primarily contains examples whose target responses were not automatically generated.
• Data covering the SQA, FC, and SER tasks.
• Long-audio examples for the ASR, ST, SS, and SQA tasks. This subset contains audio samples with durations of up to 22 minutes.

Evaluation

ASR

The model was evaluated on publicly available datasets, including both clean and noisy speech. The evaluation assessed the model's ability to transcribe audio speech into text. The word error rate (WER) is used as a metric to measure transcription accuracy . Velvet2-3B-Speech’s results were compared with other widely used models of comparable size, non ASR specialized, showing that its accuracy is on par. This demonstrates that Velvet2-3B-Speech is a highly competitive option within small-size Speech LLMs.

English ASR

Multilingual ASR

The WERs reported in the table are WERs averaged over the test datasets in the ASR Multilingual Leaderboard (Fleurs, MCV and Multilingual Librispeech)

SQQA

Spoken Query Question Answering (SQQA) is a question-answering task in which the system receives a spoken (audio) question, optionally accompanied by a textual context, and needs to generate an accurate open-ended answer.

The evaluation was performed using an LLM-as-a-judge framework. In this setup, an external LLM assesses the responses generated by Velvet2-3B-Speech according to predefined evaluation prompts. The evaluation process consists of two stages:

1. Inference, where the model generates a textual answer to an audio-based question, optionally based on a textual context.
2. Response evaluation, where the generated answer is assessed with respect to the input question.

The LLM-as-judge was prompted to assess the coherence and relevance of the generated response with respect to the question. It assigned a coherence score from 1 to 10 and provided a brief rationale for the assigned score, facilitating the analysis of borderline or ambiguous cases. The final score is reported as the average of the coherence scores.

Two evaluation settings were considered: preservation evaluation and comparative evaluation.

The purpose of the preservation evaluation is to determine whether Velvet2-3B-Speech retains the question-answering and instruction-following capabilities of Velvet2-3B across a variety of topics. To this end, the ExpertQA benchmark was used to evaluate Velvet2-3B, while replicated ExpertQA datasets containing Text-to-Speech (TTS) synthesized questions were used to evaluate Velvet2-3B-Speech.

The table below reports the coherence scores achieved by Velvet2-3B-Speech, normalized with respect to the corresponding Velvet2-3B scores (i.e., ( Velvet2-3B-Speech average score) / ( Velvet2-3B average score) * 100).

The table below shows the comparative evaluation where the average coherence scores of Velvet2-3B-Speech and other 2 well known Speech LLMs on different testsets. In HeySquad the questions are real audio clips, while in the other datasets they were TTS generated using XTTS-v2.

Speech Summarization

To evaluate the speech summarization task, we created extended-audio versions of selected test sets from VoxPopuli and MLS datasets by concatenating speech segments to obtain audio samples of up to 22 minutes in duration.

Performance was measured using the ROUGE-L metric. Following a standard practice, reference summaries were LLM-generated from the corresponding audio transcriptions and were subsequently used as ground truth for evaluation.

SER

The Speech Emotion Recognition task is evaluated by assessing a set of audio files containing speech and prompting the model to classify the emotion present in each file. The target classes are anger, happiness, sadness, and neutrality. The evaluation metric is F-score.

We cannot provide a comparative evaluation, as most Speech LLMs are not trained for this capability. Velvet2-3B-Speech achieves 0.84 and 0.86 F-score on the test splits of the CREMA-D (English) and EMOVO (Italian) datasets, respectively.

Comparison with Velvet1.5-2B-Speech

Velvet2-3B-Speech is the successor to Velvet1.5-2B-Speech, extending language coverage from 2 to 6 languages while both improving existing capabilities and introducing new ones. Existing tasks have been enhanced through advances in modeling, training data and training methodology. New tasks include speech summarization and speech question answering.

To quantify the improvements on shared capabilities, we compare the two models on ASR and SER benchmarks.

ASR (WERs)

SER (F-scores)

Robustness to prompt wording

Robustness to prompt wording is essential for real-world usability, as users may request the same task using significantly different instructions while expecting consistent behavior from the model.

To assess robustness to prompt wording variations, we considered three sets of prompts specific to the ASR task:  

• VS prompts: 30 prompts used during VelvetSpeech fine-tuning. Alternative prompt formulations were generated with ChatGPT starting from an initial seed prompt. These prompts serve as a reference condition representing a favorable, in-domain prompting scenario for Velvet2-3B-Speech. Therefore, the corresponding error rates can be interpreted as a lower bound on the model's ASR error under prompt wording variations.
• MCIF prompts: 10 prompts from the Multimodal Crosslingual Instruction-Following Benchmark benchmark. For languages other than English, prompt translations were generated using ChatGPT.
• DOWIS prompts: 10 prompts from the DOWIS prompt dataset.

The analysis covered English, Italian and Spanish languages. The evaluations were carried out on randomly sampled 1h subsets of standard ASR test sets (English: AMI, FLEURS, LibriSpeech, VoxPopuli; Spanish: FLEURS, MCV, MLS; Italian: FLEURS, MCV, MLS). The table shows the average WER across languages and test sets.

Note: Voxtral-Mini-3B-2507 model provides a dedicated transcription mode, which was not used in this evaluation. Instead, the evaluation was designed to assess the model's ability to infer the requested task solely from the provided prompt.

The results indicate that Velvet2-3B-Speech is considerably more robust to prompt wording variations than the other evaluated models, at least for what concerns the ASR task.

Usage

Velvet2-3B-Speech can be served with:

Transformers

import torch
import librosa

from transformers import AutoModel, AutoProcessor
from huggingface_hub import hf_hub_download

device = "cuda" if torch.cuda.is_available() else "cpu"

base_model = "AlmawaveLabs/Velvet2-3B-Speech"

# Load the model
processor = AutoProcessor.from_pretrained(base_model, trust_remote_code=True)
model = AutoModel.from_pretrained(base_model,
                                  trust_remote_code=True,
                                  use_safetensors=True,
                                  low_cpu_mem_usage=True,
                                  device_map="auto",
                                  torch_dtype=torch.bfloat16)
model.to(device)

# Input text
user_prompt = "Transcribe the following audio"
# Input audio
audio_path = hf_hub_download(repo_id=base_model, filename="audio_examples/1188-133604-0040.flac")
# Load audio
audio = librosa.load(audio_path, sr=processor.feature_extractor.sampling_rate)[0]

# Create the text prompt
messages = []
messages.append({"role": "user", "content": [
                    {"type": "text", "text": user_prompt},
                    {"type": "audio", "audio_url": audio_path}
                  ]})
prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

# Run the processor
model_inputs = processor(text=prompt,
                         audios=audio,
                         return_tensors="pt",
                         padding=True,
                         sampling_rate=processor.feature_extractor.sampling_rate,
                         return_token_type_ids=False,
                         add_special_tokens=False).to(device)

# Generate Output 
model_outputs = model.generate(**model_inputs, max_new_tokens=256, use_cache=True)
model_outputs = model_outputs[:, model_inputs.input_ids.size(1):]
output_text = processor.batch_decode(model_outputs,
                                     skip_special_tokens=True,
                                     clean_up_tokenization_spaces=False)

print(f"Velvet2-3B-Speech Output: {output_text[0]}")

Tool Calling

The processor's native chat template injects the supplied tools. No parser plugin is needed. In this path, tool definitions use the function-schema form accepted by processor.apply_chat_template, which includes the OpenAI API's outer {"type": "function", "function": ...} wrapper.

import ast
import re

import torch
import librosa

from transformers import AutoProcessor, AutoModel
from huggingface_hub import hf_hub_download

base_model = "AlmawaveLabs/Velvet2-3B-Speech"

tools = [
    {
    "type": "function",
    "function": {
        "name": "get_weather_forecast",
        "description": "Get the weather forecast for a city on a given date.",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "City name."},
                "date": {
                    "type": "string",
                    "description": "Date in YYYY-MM-DD format.",
                },
            },
            "required": ["location", "date"],
        },
      }
    }
]

def get_weather_forecast(location, date):
    # Mock implementation; replace with your own tool.
    return {
        "location": location,
        "date": date,
        "forecast": "Sunny, 18°C (64.4°F)",
    }


available_tools = {"get_weather_forecast": get_weather_forecast}

# Load the model
processor = AutoProcessor.from_pretrained(base_model, trust_remote_code=True)
model = AutoModel.from_pretrained(base_model,
                                  trust_remote_code=True,
                                  use_safetensors=True,
                                  low_cpu_mem_usage=True,
                                  device_map="auto",
                                  torch_dtype=torch.bfloat16)
model.to(device)


def generate(messages, audio):
    prompt = processor.apply_chat_template(messages, tools=tools, add_generation_prompt=True, tokenize=False)

    # Run the processor
    model_inputs = processor(text=prompt,
                             audios=audio,
                             return_tensors="pt",
                             padding=True,
                             sampling_rate=processor.feature_extractor.sampling_rate,
                             return_token_type_ids=False,
                             add_special_tokens=False).to(device)
    
    output_ids = model.generate(**model_inputs, max_new_tokens=256, use_cache=True)
  
    # Preserve the special tool-call markers while decoding.
    text = processor.decode(
        output_ids[0, model_inputs.input_ids.shape[1]:],
        skip_special_tokens=False,
    )
    return text.removesuffix(processor.tokenizer.eos_token).strip()


# Input audio
audio_path = hf_hub_download(repo_id=model_name, filename="audio_examples/tools_test.flac")
# Load audio
audio = librosa.load(audio_path, sr=processor.feature_extractor.sampling_rate)[0]

messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio_url": audio_path},
        ]
    },
]

response = generate(messages)
match = re.search(r"<tool_call>(.*?)</tool_call>", response, re.DOTALL)

if match:
    tool_calls_list = ast.literal_eval(match.group(1))
    # Keep the original tool call output in the conversation.
    messages.append({"tool_calls": tool_calls_list})
    
    for tool_call in tool_calls_list:
        function = available_tools.get(tool_call["name"])
        if function is None:
            raise ValueError(f"Unknown tool: {tool_call['name']}")

        result = function(**tool_call["arguments"])
        print(f"Tool call: {tool_call['name']}({tool_call['arguments']})")
        messages.append(
            {
                "role": "tool",
                "content": str(result),
            }
        )

    response = generate(messages, audio=audio)

print(f"Velvet2-3B-Speech Output: {response}")

vLLM vLLM support for Velvet2-3B-Speech requires specific customization to handle audio input correctly. Ollama Support for GGUF format will be available soon.

Responsible AI

Almawave Labs is a signatory of the EU GPAI Code of Practice, see https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai .

Velvet2-3B-Speech incorporates safety measures throughout dataset preparation, training, and post-training stages.

These measures include data filtering, quality controls, alignment-oriented instruction tuning, and safety-focused evaluation procedures.

Despite these efforts, no language model can guarantee complete prevention of harmful, inappropriate, or inaccurate outputs. Developers are responsible for implementing application-specific safeguards and conducting appropriate risk assessments before deploying the model in production environments.

Large language models are versatile technologies designed to serve a wide range of applications. However, they are not intended to meet every developer's safety requirements out-of-the-box, as these requirements naturally vary depending on the specific use case and application context.

Safety

For our instruction-trained model, we have undertaken comprehensive exercises, engaged in adversarial internal and external evaluations, and put into place mitigation techniques to reduce risks. These exercises were designed to thoroughly examine the model's limitations and potential, simulating real and hypothetical scenarios where undesirable behavior might arise.

However, despite these efforts, it is inevitable that some residual hazards are bound to exist, as every large language model presents intrinsic complexities that cannot be completely eliminated.

Then, developers are advised to implement suitable safety measures and exercise due diligence, tailoring these safeguards to align with their product policies and the specific requirements of their applications.

Some trade-offs between model helpfulness and alignment are likely inevitable. Developers should thoughtfully balance the benefits of alignment and helpfulness for their specific applications and audiences. They must also remain aware of residual risks when using Velvet models and leverage additional safety tools as necessary to achieve an appropriate safety standard for their use case. We advise developers to carefully evaluate risks in the context of their specific use case. They should consider the potential implications of a model failure in their applications and put adequate measures in place to manage such eventualities.

In parallel, we are collaborating with the scientific and industrial community to establish AI safety benchmark standards that are transparent, rigorous, and interpretable. The goal is to promote a better understanding of the risks associated with large language models and support the development of safer and more responsible solutions.

Governance and Internal Oversight

Almawave Labs has established an internal governance framework for the management and continuous oversight of the Velvet model family. Key governance elements include:

Supervision by an Ethical and Technical Committee to ensure the model aligns with principles of transparency, fairness, and safety. Ongoing bias monitoring through auditing tools, with iterative updates to improve alignment with ethical guidelines. Restrictions on commercial and institutional usage to ensure compliance with regulatory frameworks and shared responsibility principles. Periodic review processes to assess the model’s impact in high-risk applications.

Bias, Risks and Limitations

Velvet has been trained on a dataset that, despite all the data curation efforts, might include toxic language and societal biases. This means that models in the Velvet family may reproduce these biases and produce harmful responses when prompted with such inputs. This is a common issue in AI models trained on large datasets, as they can inadvertently perpetuate the biases present in the data. Furthermore, the model may generate inaccurate, incomplete, or redundant responses, which could be socially unacceptable or undesirable, even if the input prompt is not explicitly offensive. This is a potential flaw in the model's design and training process, and it underscores the importance of careful validation and monitoring of AI systems to ensure that they are functioning as intended.

Additionally, using the recommended prompt template is crucial to mitigate the risk of harmful responses, as it is designed to guide the model towards more appropriate and safe outputs. However, it is important to note that the model's performance may still vary depending on the specific context and complexity of the input prompt.

Finally, when using this model in an agentic workflow, it is essential to validate that all imported packages and dependencies are from trusted sources to ensure the model's security and integrity. This is a critical step in maintaining the model's ethical and responsible use, and it is important to prioritize end-to-end security measures to prevent any potential vulnerabilities or breaches.

Future versions of Velvet will integrate automated red-teaming protocols, continuously stress-testing the model against adversarial prompts to identify and mitigate emerging risks.

Sensitive Data Handling and Usage Restrictions

The Velvet model has not been trained on unauthorized personal data and must not be used to process sensitive data without appropriate security measures.

Usage Restrictions:

• Prohibited use on sensitive healthcare, financial, or government data without specific safeguards.
• Mandatory human validation in scenarios where the model’s outputs could have legal or ethical consequences.
• High-risk applications (legal, medical, public governance) must implement content filtering and auditing techniques to ensure response quality and safety.

Ethical Considerations

Almawave Labs core values are openness, inclusivity, and helpfulness. We aim to create AI that is accessible and beneficial for everyone, regardless of their background. Velvet models are designed to be respectful of diverse perspectives and avoid unnecessary judgments. Therefore, Velvet models are designed to be inclusive and respectful of diverse perspectives and needs. We strive to avoid unnecessary judgment or the imposition of normative views, recognizing that content deemed problematic in some contexts can have valuable applications in others.

We deeply respect the dignity and autonomy of all users, particularly their right to free thought and expression, which are fundamental to innovation and progress.

While we have taken significant steps to ensure the safety and reliability of Velvet models, it is important to acknowledge that they may occasionally generate inaccurate, biased, or unsafe responses.

Almawave Labs is actively engaging with ethics committees and domain experts to ensure continuous oversight of Velvet’s outputs, improving safeguards through community feedback.

We strongly encourage the community to exercise caution and conduct thorough safety testing and fine-tuning when using Velvet models for specific tasks.

Opinions expressed by Velvet depend on training data and do not reflect any opinions of Almawave Labs.

Contributions

In addition to contributors to Velvet2-3B

Speech team: Leonardo Badino, Fabio Brugnara, Giovanni Morrone, Martina Valente, Enrico Zovato.

Direction: Andrea Favalli, Raniero Romagnoli.