English to Urdu Neural Machine Translation using Transformer with Averaged Word Embeddings

Abstract

In this era of computers, the World Wide Web (WWW) content is vital for everyone. Most of the useful content on the web is in English. Machine translation from the web content into the national language of Pakistan, i.e. Urdu, has several applications such as Urdu text generation, language resources creation, language research and availability of knowledge to illiterate individuals who cannot comprehend English but know Urdu. We propose a novel English-to-Urdu machine translation model in this research work, based on the transformer model that exploits the average of three word embeddings. These three word embeddings are Urdu word2vec (skipgram-based), part-of-speech-ngram (POS-Ngram) embeddings, and POS-POS embeddings, both of which encode the rich morphological and morphosyntactic features of Urdu language inside the word embeddings. Experiments are performed using a manually compiled English-Urdu parallel corpus from OPUS corpora and Github. The proposed transformer-based approach is compared to fine-tuned Llama-3-8B, T5-small, Long Short-Term Memory (LSTM), and Bi-directional LSTM (Bi-LSTM). The evaluation metrics used are BLEU and ROUGE-L scores. The results suggest that the proposed model outperforms T5-small, LSTM and Bi-LSTM by ≈2.15, 7.44 and 5.53 points respectively, in BLEU score and by ≈1.7, 2.5 and 4.12 points, respectively, in ROUGE-L score. The proposed model shows comparable performance to the fine-tuned Llama-3-8B.