Ha van ötlet, hogy még milyen adatbázist vegyek be a tesztbe, akkor írjátok meg.
Összehasonlító táblázatok adatbások szerint
mozilla-foundation/common_voice_17_0
A sarpba/whisper-large-v2-CV18-hu-cleaned a CV-18 teljes tisztított adatbázison van kiképezve, az eredménye CV-17-en emiatt nem releváns!
A whisper-large-turbo nagyon durván, a whisper-large, whisper-large-v2, sarpba/whisper-hu-small-finetuned halucinálni kezdett, ez torzítja a végeredményt, ez okozza a fura táblázatot.
KTH/hungarian-single-speaker-tts
Összesített táblázat
| model_name | WER | CER | Norm WER | Norm CER | dataset | batch size | test lang | runtime |
|---|---|---|---|---|---|---|---|---|
| benmajor27/whisper-large-v3-hu_full | 9.42 | 1.85 | 7.80 | 1.50 | CV_17_0_hu_test | 32 | hu | 3618.87 |
| sarpba/whisper-hu-large-v3-turbo-finetuned | 12.67 | 2.42 | 10.59 | 1.95 | CV_17_0_hu_test | 32 | hu | 4458.90 |
| benmajor27/whisper-large-v3-hu_full | 20.54 | 6.30 | 13.58 | 5.01 | g_fleurs_test_hu | 16 | hu | 498.56 |
| sarpba/whisper-hu-large-v3-turbo-finetuned | 29.04 | 6.04 | 13.95 | 2.85 | KTH_hun_test | 32 | hu | 1854.34 |
| sarpba/whisper-hu-large-v3-turbo-finetuned | 18.69 | 6.98 | 14.03 | 6.20 | voxpopuli_hu_test | 32 | hu | 481.67 |
| benmajor27/whisper-large-v3-hu_full | 19.81 | 7.35 | 14.07 | 6.36 | voxpopuli_hu_test | 16 | hu | 667.62 |
| openai/whisper-large-v3 | 19.77 | 4.81 | 14.62 | 3.73 | g_fleurs_test_hu | 16 | hu | 617.91 |
| sarpba/whisper-large-v2-CV18-hu-cleaned | 20.34 | 7.45 | 14.63 | 6.41 | voxpopuli_hu_test | 16 | hu | 693.03 |
| sarpba/whisper-large-v2-CV18-hu-cleaned | 21.88 | 6.04 | 14.75 | 4.65 | g_fleurs_test_hu | 16 | hu | 477.16 |
| sarpba/whisper-hu-base-finetuned-V2 | 35.47 | 6.92 | 15.66 | 3.58 | CV_17_0_hu_test | 32 | hu | 372.19 |
| sarpba/whisper-large-v2-CV18-hu-cleaned | 29.08 | 6.70 | 15.96 | 3.35 | KTH_hun_test | 16 | hu | 1962.98 |
| openai/whisper-large-v3-turbo | 21.09 | 5.04 | 16.05 | 4.00 | g_fleurs_test_hu | 32 | hu | 364.72 |
| sarpba/whisper-hu-small-finetuned | 31.60 | 6.92 | 16.23 | 3.55 | KTH_hun_test | 32 | hu | 601.23 |
| sarpba/whisper-hu-large-v3-turbo-finetuned | 22.07 | 7.00 | 16.26 | 5.90 | g_fleurs_test_hu | 32 | hu | 396.84 |
| openai/whisper-large-v3 | 27.25 | 6.16 | 16.29 | 3.38 | KTH_hun_test | 32 | hu | 2614.78 |
| openai/whisper-large-v3 | 20.94 | 9.14 | 16.47 | 8.41 | voxpopuli_hu_test | 16 | hu | 826.51 |
| openai/whisper-large-v3-turbo | 27.61 | 6.32 | 17.28 | 3.63 | KTH_hun_test | 32 | hu | 1719.94 |
| sarpba/whisper-hu-small-finetuned | 21.03 | 4.52 | 17.34 | 3.68 | CV_17_0_hu_test | 32 | hu | 1207.23 |
| sarpba/whisper-base-hungarian_v1 | 23.49 | 8.17 | 17.70 | 7.09 | voxpopuli_hu_test | 32 | hu | 92.13 |
| openai/whisper-large-v3 | 21.81 | 5.81 | 18.07 | 4.95 | CV_17_0_hu_test | 16 | hu | 5676.63 |
| benmajor27/whisper-large-v3-hu_full | 30.93 | 6.48 | 18.35 | 3.33 | KTH_hun_test | 32 | hu | 1796.73 |
| sarpba/whisper-hu-small-finetuned | 25.27 | 6.50 | 19.22 | 5.27 | g_fleurs_test_hu | 32 | hu | 154.49 |
| openai/whisper-large-v2 | 24.04 | 6.24 | 19.26 | 5.15 | g_fleurs_test_hu | 16 | hu | 627.70 |
| openai/whisper-large-v3-turbo | 23.03 | 5.70 | 19.45 | 4.85 | CV_17_0_hu_test | 32 | hu | 4179.45 |
| sarpba/whisper-hu-base-finetuned-V2 | 41.87 | 11.02 | 19.60 | 5.91 | KTH_hun_test | 32 | hu | 220.65 |
| sarpba/whisper-base-hungarian_v1 | 35.58 | 8.32 | 19.94 | 4.81 | KTH_hun_test | 32 | hu | 259.39 |
| openai/whisper-large-v2 | 32.91 | 7.66 | 19.99 | 4.43 | KTH_hun_test | 32 | hu | 2776.37 |
| sarpba/whisper-hu-tiny-finetuned-V2 | 38.74 | 8.14 | 20.05 | 4.84 | CV_17_0_hu_test | 32 | hu | 284.77 |
| sarpba/whisper-hu-base-finetuned-V2 | 31.56 | 11.04 | 20.37 | 9.09 | voxpopuli_hu_test | 32 | hu | 90.88 |
| sarpba/whisper-hu-tiny-finetuned-V2 | 44.09 | 10.54 | 21.05 | 5.15 | KTH_hun_test | 32 | hu | 173.14 |
| sarpba/whisper-hu-tiny-finetuned | 27.59 | 9.69 | 21.35 | 8.18 | voxpopuli_hu_test | 32 | hu | 69.30 |
| openai/whisper-large-v2 | 25.97 | 6.57 | 21.82 | 5.47 | CV_17_0_hu_test | 16 | hu | 9275.54 |
| openai/whisper-large | 33.45 | 7.77 | 22.38 | 4.85 | KTH_hun_test | 32 | hu | 2758.02 |
| sarpba/whisper-hu-tiny-finetuned-V2 | 33.81 | 10.83 | 22.50 | 8.71 | voxpopuli_hu_test | 32 | hu | 62.08 |
| sarpba/whisper-hu-small-finetuned | 28.62 | 11.84 | 23.48 | 11.17 | voxpopuli_hu_test | 32 | hu | 223.89 |
| sarpba/whisper-base-hungarian_v1 | 27.65 | 6.77 | 23.53 | 5.77 | CV_17_0_hu_test | 32 | hu | 460.27 |
| sarpba/whisper-hu-tiny-finetuned | 39.55 | 10.02 | 23.61 | 6.38 | KTH_hun_test | 32 | hu | 195.42 |
| openai/whisper-large | 30.64 | 13.95 | 24.83 | 12.98 | voxpopuli_hu_test | 16 | hu | 941.59 |
| openai/whisper-medium | 31.10 | 12.81 | 25.43 | 11.77 | voxpopuli_hu_test | 32 | hu | 606.56 |
| sarpba/whisper-hu-base-finetuned-V2 | 38.62 | 13.25 | 26.06 | 10.75 | g_fleurs_test_hu | 32 | hu | 62.17 |
| openai/whisper-large | 30.13 | 8.93 | 26.20 | 8.04 | CV_17_0_hu_test | 16 | hu | 5909.03 |
| openai/whisper-large-v2 | 31.06 | 16.10 | 26.22 | 15.03 | voxpopuli_hu_test | 16 | hu | 931.83 |
| openai/whisper-medium | 37.04 | 8.85 | 26.28 | 5.88 | KTH_hun_test | 32 | hu | 1479.77 |
| sarpba/whisper-hu-tiny-finetuned | 30.81 | 7.67 | 26.63 | 6.60 | CV_17_0_hu_test | 32 | hu | 328.25 |
| openai/whisper-large | 31.74 | 10.69 | 26.67 | 9.57 | g_fleurs_test_hu | 16 | hu | 711.97 |
| openai/whisper-medium | 33.04 | 9.93 | 27.97 | 8.34 | g_fleurs_test_hu | 32 | hu | 450.89 |
| sarpba/whisper-base-hungarian_v1 | 37.16 | 11.96 | 30.60 | 10.43 | g_fleurs_test_hu | 32 | hu | 67.86 |
| openai/whisper-medium | 34.46 | 9.12 | 30.63 | 8.05 | CV_17_0_hu_test | 32 | hu | 3317.29 |
| sarpba/whisper-hu-tiny-finetuned-V2 | 42.58 | 14.92 | 31.09 | 12.61 | g_fleurs_test_hu | 32 | hu | 49.01 |
| sarpba/whisper-hu-tiny-finetuned | 40.32 | 12.85 | 33.99 | 11.33 | g_fleurs_test_hu | 32 | hu | 51.74 |
| openai/whisper-small | 50.58 | 13.32 | 41.63 | 10.43 | KTH_hun_test | 32 | hu | 597.42 |
| openai/whisper-small | 50.07 | 15.69 | 45.54 | 14.40 | g_fleurs_test_hu | 32 | hu | 185.89 |
| openai/whisper-small | 57.51 | 24.38 | 51.52 | 23.59 | voxpopuli_hu_test | 32 | hu | 273.61 |
| openai/whisper-small | 55.67 | 16.77 | 52.20 | 15.62 | CV_17_0_hu_test | 32 | hu | 1398.06 |
| openai/whisper-base | 70.75 | 24.18 | 64.75 | 21.40 | KTH_hun_test | 32 | hu | 293.91 |
| openai/whisper-base | 75.37 | 31.70 | 70.56 | 30.51 | voxpopuli_hu_test | 32 | hu | 120.48 |
| openai/whisper-tiny | 86.44 | 34.23 | 82.86 | 31.67 | KTH_hun_test | 32 | hu | 228.43 |
| openai/whisper-base | 89.82 | 40.00 | 86.61 | 37.75 | g_fleurs_test_hu | 32 | hu | 118.69 |
| openai/whisper-base | 95.66 | 39.98 | 93.67 | 38.51 | CV_17_0_hu_test | 32 | hu | 779.32 |
| openai/whisper-tiny | 108.61 | 58.69 | 106.29 | 55.98 | g_fleurs_test_hu | 32 | hu | 90.65 |
| openai/whisper-tiny | 110.67 | 48.06 | 106.81 | 45.99 | voxpopuli_hu_test | 32 | hu | 101.88 |
| openai/whisper-large-v3-turbo | 111.51 | 71.77 | 108.92 | 71.77 | voxpopuli_hu_test | 32 | hu | 494.32 |
| openai/whisper-tiny | 120.86 | 55.10 | 119.12 | 53.19 | CV_17_0_hu_test | 32 | hu | 597.92 |