w2v-bert-2.0-hungarian-CV24.0

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the Common Voice Scripted Speech 24.0 - Hungarian dataset. It achieves the following results on the evaluation set:

Loss: 0.1203
Wer: 0.0781

Evaluation data

Results on the Common Voice Scripted Speech 24.0 - Hungarian test dataset

================================================================================ EVALUATION RESULTS SUMMARY

                      Metric  Value    Std
       Word Error Rate (WER)  5.65% 11.90%
  Character Error Rate (CER)  1.47%  4.23%
   Sentence Error Rate (SER) 32.26% 46.75%
      Match Error Rate (MER)  5.42% 10.49%
 Word Information Lost (WIL)  8.50% 15.24%

Word Information Preserved (WIP) 91.50% 15.24%

            Total Samples  12066

================================================================================ SAMPLE ERRORS (Highest WER)

--- Sample 1 (WER: 450.00%, CER: 254.55%) --- Reference: chet greene Hypothesis: chet|gren|egy|ilyen|tisztázás|nek|áa|| Insertions: 28 Deletions: 0 Substitutions: 0

--- Sample 2 (WER: 150.00%, CER: 14.29%) --- Reference: radkersburg ut Hypothesis: ratkersburg|út Insertions: 1 Deletions: 0 Substitutions: 1

--- Sample 3 (WER: 150.00%, CER: 33.33%) --- Reference: gyolcsot sző Hypothesis: gyolcsat|szün Insertions: 2 Deletions: 0 Substitutions: 2

--- Sample 4 (WER: 133.33%, CER: 14.81%) --- Reference: universala esperanto asocio Hypothesis: universala|eszperanto|a|szócio Insertions: 4 Deletions: 0 Substitutions: 0

--- Sample 5 (WER: 100.00%, CER: 25.00%) --- Reference: kotródj Hypothesis: kotrudj Insertions: 0 Deletions: 1 Substitutions: 1

Use

Due to tokenizer metadata resolution inconsistencies in the HuggingFace hub loader, users are recommended to download the repository locally and perform inference using local_files_only=True.

git lfs install
git clone https://huggingface.co/GaborMadarasz/w2v-bert-2.0-hungarian-CV24.0

from transformers import AutoProcessor, AutoModelForCTC

model_path = "./w2v-bert-2.0-hungarian-CV24.0"

processor = AutoProcessor.from_pretrained(
    model_path,
    local_files_only=True
)

model = AutoModelForCTC.from_pretrained(
    model_path,
    local_files_only=True
)

model.eval()

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 2
eval_batch_size: 2
seed: 42
gradient_accumulation_steps: 8
total_train_batch_size: 16
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
lr_scheduler_warmup_ratio: 0.1
num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
0.1555	0.2838	1000	0.1763	0.1785
0.1198	0.5676	2000	0.1773	0.2019
0.1324	0.8514	3000	0.1679	0.1745
0.1358	1.1351	4000	0.1635	0.1649
0.1161	1.4189	5000	0.1486	0.1566
0.1027	1.7027	6000	0.1512	0.1622
0.0897	1.9864	7000	0.1359	0.1457
0.0706	2.2702	8000	0.1328	0.1348
0.0785	2.5540	9000	0.1341	0.1324
0.066	2.8377	10000	0.1255	0.1335
0.0638	3.1215	11000	0.1092	0.1196
0.0529	3.4053	12000	0.1119	0.1108
0.0567	3.6890	13000	0.1167	0.1084
0.0533	3.9728	14000	0.1011	0.1039
0.0471	4.2565	15000	0.1072	0.1066
0.0337	4.5403	16000	0.1189	0.1052
0.0338	4.8241	17000	0.1029	0.1113
0.0346	5.1078	18000	0.0970	0.1004
0.0264	5.3916	19000	0.1119	0.1082
0.0263	5.6754	20000	0.1194	0.0946
0.0231	5.9592	21000	0.0981	0.0978
0.0196	6.2429	22000	0.1038	0.0994
0.0228	6.5267	23000	0.0996	0.0948
0.013	6.8105	24000	0.1023	0.0938
0.015	7.0942	25000	0.1016	0.0876
0.0124	7.3780	26000	0.1088	0.0903
0.0128	7.6618	27000	0.1027	0.0895
0.0111	7.9456	28000	0.1064	0.0844
0.0039	8.2293	29000	0.1186	0.0844
0.007	8.5131	30000	0.1120	0.0855
0.0062	8.7969	31000	0.1060	0.0836
0.0041	9.0806	32000	0.1114	0.0788
0.004	9.3644	33000	0.1193	0.0812
0.0048	9.6482	34000	0.1183	0.0773
0.0029	9.9320	35000	0.1203	0.0781

Framework versions

Transformers 4.57.3
Pytorch 2.9.0+cu128
Datasets 4.5.0
Tokenizers 0.22.1

Contact

gabor.madarasz@gmail.com