whisper-large-v3-turbo-FLEURS-GL-EN

This model is a fine-tuned version of openai/whisper-large-v3-turbo trained on juanjucm/FLEURS-SpeechT-GL-EN for Galician-to-English Text to Speech Translation task. It takes galician speech audios as input and generates the correspondant translated transcription in English.

The motivation behind this work is to increase the visibility of the Galician language, making it more accessible for non-Galician speakers to understand and engage with Galician audio content.

This model was developed during a 3-week Speech Translation workshop organised by Yasmin Moslem.

Performance and training details

Baseline model achieved a BLEU score of 5.0 on the evaluation dataset.

After fine-tuning, it achieves the following results on the evaluation set:

• Loss: 1.4958
• Wer: 71.6814
• BLEU: 18.9665

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• total_train_batch_size: 32
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 4
• mixed_precision_training: Native AMP

Training results

We used BLEU Score as our reference translation metric for selecting the best checkpoint after training.

Training Loss	Epoch	Step	Validation Loss	Wer	Bleu
4.2751	1.0	5	3.8850	76.6962	18.0512
2.3984	2.0	10	2.6965	97.0501	13.5327
1.4958	3.0	15	2.2308	71.6814	18.9665
1.27	4.0	20	2.0454	128.3186	12.2446

Framework versions

• Transformers 4.45.1
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.20.0