gemma-4-E2B-it-CV24_hun

Fine-tuned google/gemma-4-E2B-it for Hungarian automatic speech recognition (ASR) using LoRA adapters, trained with Unsloth.

Model Description

This model is a LoRA fine-tune of Gemma 4 E2B (2B parameter Mixture-of-Experts multimodal variant), adapted for Hungarian speech transcription using the multimodal chat format. The audio encoder and language model backbone are fine-tuned jointly via low-rank adaptation.

• Base model: google/gemma-4-E2B-it
• Architecture: Gemma 4 multimodal (text + audio)
• Fine-tuning method: LoRA (r=16, alpha=16) via Unsloth FastModel
• Saved format: Merged float16 weights (via push_to_hub_merged)

Evaluation

On 448 held-out random sample

Training Details

Dataset

• Training data: Mozilla common Voice 24 Hungarian split + Voxpopuli Hungarian split + FLEURS Hungarian split
• Dataset size: 88547 audio-text pairs used for training; 448 held-out random sample for evaluation
• Audio sampling rate: 16,000 Hz

LoRA Configuration

Training Hyperparameters

Data Formatting

Each sample is formatted into a multimodal chat conversation: <|turn>system You are an assistant that transcribes speech accurately.<turn|> <|turn>user <|audio|>Please transcribe this audio.<turn|> <|turn>model [transcription]<turn|>

Training Hardware

Single RTX3060 12GB GPU

598.12 minutes used for training.

Peak reserved memory = 9.73 GB.

Usage

from unsloth import FastModel
from transformers import TextStreamer

model, processor = FastModel.from_pretrained(
    model_name="GaborMadarasz/gemma-4-E2B-it-CV24_hun",
    max_seq_length=1024,
    load_in_4bit=True,
)

def transcribe(audio_array):
    messages = [
        {
            "role": "system",
            "content": [{"type": "text", "text": "You are an assistant that transcribes speech accurately."}],
        },
        {
            "role": "user",
            "content": [
                {"type": "audio", "audio": audio_array},
                {"type": "text", "text": "Please transcribe this audio."}
            ]
        }
    ]
    model.generate(
        **processor.apply_chat_template(
            messages,
            add_generation_prompt=True,
            tokenize=True,
            return_dict=True,
            return_tensors="pt",
        ).to("cuda"),
        max_new_tokens=256,
        do_sample=False,
        streamer=TextStreamer(processor, skip_prompt=True),
    )

Note: Audio input must be resampled to 16,000 Hz before passing to the processor.

Limitations

• Optimized specifically for Hungarian speech; performance on other languages is not guaranteed.
• Trained on 1 epoch of Common Voice, Voxpopuli, FLEURS HU data; rare vocabulary, dialects, and heavy accents may degrade accuracy.
• The model was fine-tuned with load_in_4bit=True; the merged float16 weights are provided for full-precision inference.

Framework

• Unsloth — efficient LoRA fine-tuning
• TRL SFTTrainer + UnslothVisionDataCollator
• transformers==5.5.0