Benchmarking Procedural Language Understanding for Low-Resource Languages: A Case Study on Turkish
Uzunoglu A, Şahin GG (2023)
Publication Type: Conference contribution
Publication year: 2023
Publisher: Association for Computational Linguistics (ACL)
Book Volume: 1
Pages Range: 804-819
Conference Proceedings Title: Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: Long Papers, IJCNLP-AACL 2023
Event location: Bali, IDN
ISBN: 9798891760134
DOI: 10.18653/v1/2023.ijcnlp-main.52
Abstract
Understanding procedural natural language (e.g., step-by-step instructions) is a crucial step to execution and planning. However, while there are ample corpora and downstream tasks available in English, the field lacks such resources for most languages. To address this gap, we conduct a case study on Turkish procedural texts. We first expand the number of tutorials in Turkish wikiHow from 2,000 to 52,000 using automated translation tools, where the translation quality and loyalty to the original meaning are validated by a team of experts on a random set. Then, we generate several downstream tasks on the corpus, such as linking actions, goal inference, and summarization. To tackle these tasks, we implement strong baseline models via fine-tuning large language-specific models such as TR-BART and BERTurk, as well as multilingual models such as mBART, mT5, and XLM. We find that language-specific models consistently outperform their multilingual models by a significant margin across most procedural language understanding (PLU) tasks. We release our corpus, downstream tasks and the baseline models with https://github.com/GGLAB-KU/turkish-plu.
Involved external institutions
United States (USA) (US)
Turkey (TR)How to cite
APA:
Uzunoglu, A., & Şahin, G.G. (2023). Benchmarking Procedural Language Understanding for Low-Resource Languages: A Case Study on Turkish. In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, Adila Alfa Krisnadhi (Eds.), Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: Long Papers, IJCNLP-AACL 2023 (pp. 804-819). Bali, IDN: Association for Computational Linguistics (ACL).
MLA:
Uzunoglu, Arda, and Gözde Gül Şahin. "Benchmarking Procedural Language Understanding for Low-Resource Languages: A Case Study on Turkish." Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, IJCNLP-AACL 2023, Bali, IDN Ed. Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, Adila Alfa Krisnadhi, Association for Computational Linguistics (ACL), 2023. 804-819.
BibTeX: Download