gemma-4-E2B-it-CV24_hun

Fine-tuned google/gemma-4-E2B-it for Hungarian automatic speech recognition (ASR) using LoRA adapters, trained with Unsloth.

Model Description

This model is a LoRA fine-tune of Gemma 4 E2B (2B parameter Mixture-of-Experts multimodal variant), adapted for Hungarian speech transcription using the multimodal chat format. The audio encoder and language model backbone are fine-tuned jointly via low-rank adaptation.

Base model: google/gemma-4-E2B-it
Architecture: Gemma 4 multimodal (text + audio)
Fine-tuning method: LoRA (r=16, alpha=16) via Unsloth FastModel
Saved format: Merged float16 weights (via push_to_hub_merged)

Evaluation

On 448 held-out random sample

Metric	Value
Test WER	0.2009 (20.09%)
Test CER	0.0583 (5.83%)

Training Details

Dataset

Training data: Mozilla common Voice 24 Hungarian split + Voxpopuli Hungarian split + FLEURS Hungarian split
Dataset size: 88547 audio-text pairs used for training; 448 held-out random sample for evaluation
Audio sampling rate: 16,000 Hz

LoRA Configuration

Parameter	Value
`r`	16
`lora_alpha`	16
`lora_dropout`	0
`bias`	none
`use_rslora`	False
Target modules (language)	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
Target modules (audio)	`post`, `linear_start`, `linear_end`, `embedding_projection`, `ffw_layer_1`, `ffw_layer_2`, `output_proj`
`finetune_vision_layers`	False
`finetune_language_layers`	True
`finetune_attention_modules`	True
`finetune_mlp_modules`	True

Training Hyperparameters

Parameter	Value
`per_device_train_batch_size`	2
`gradient_accumulation_steps`	4
`effective_batch_size`	8
`num_train_epochs`	1
`learning_rate`	8e-5
`lr_scheduler_type`	cosine
`warmup_ratio`	0.03
`optimizer`	adamw_8bit
`weight_decay`	0.001
`max_seq_length`	1024
`seed`	3407
`load_in_4bit`	True (training-time quantization)

Data Formatting

Training Hardware

Single RTX3060 12GB GPU

598.12 minutes used for training.

Peak reserved memory = 9.73 GB.

Usage

from unsloth import FastModel
from transformers import TextStreamer

model, processor = FastModel.from_pretrained(
    model_name="GaborMadarasz/gemma-4-E2B-it-CV24_hun",
    max_seq_length=1024,
    load_in_4bit=True,
)

def transcribe(audio_array):
    messages = [
        {
            "role": "system",
            "content": [{"type": "text", "text": "You are an assistant that transcribes speech accurately."}],
        },
        {
            "role": "user",
            "content": [
                {"type": "audio", "audio": audio_array},
                {"type": "text", "text": "Please transcribe this audio."}
            ]
        }
    ]
    model.generate(
        **processor.apply_chat_template(
            messages,
            add_generation_prompt=True,
            tokenize=True,
            return_dict=True,
            return_tensors="pt",
        ).to("cuda"),
        max_new_tokens=256,
        do_sample=False,
        streamer=TextStreamer(processor, skip_prompt=True),
    )

Note: Audio input must be resampled to 16,000 Hz before passing to the processor.

Limitations

Optimized specifically for Hungarian speech; performance on other languages is not guaranteed.
Trained on 1 epoch of Common Voice, Voxpopuli, FLEURS HU data; rare vocabulary, dialects, and heavy accents may degrade accuracy.
The model was fine-tuned with load_in_4bit=True; the merged float16 weights are provided for full-precision inference.

Framework

Unsloth — efficient LoRA fine-tuning
TRL SFTTrainer + UnslothVisionDataCollator
transformers==5.5.0

GaborMadarasz/gemma-4-E2B-it-CV24-VP-FL_hun