Metric-Based In-context Learning: A Case Study in Text Simplification
Vadlamannati S, Şahin GG (2023)
Publication Type: Conference contribution
Publication year: 2023
Publisher: Association for Computational Linguistics (ACL)
Pages Range: 253-268
Conference Proceedings Title: INLG 2023 - 16th International Natural Language Generation Conference, Proceedings of the Conference
Event location: Prague, CZE
ISBN: 9798891760011
DOI: 10.18653/v1/2023.inlg-main.18
Abstract
In-context learning (ICL) for large language models has proven to be a powerful approach for many natural language processing tasks. However, determining the best method to select examples for ICL is nontrivial as the results can vary greatly depending on the quality, quantity, and order of examples used. In this paper, we conduct a case study on text simplification (TS) to investigate how to select the best and most robust examples for ICL. We propose Metric-Based in-context Learning (MBL) method that utilizes commonly used TS metrics such as SARI, compression ratio, and BERT-Precision for selection. Through an extensive set of experiments with various-sized GPT models on standard TS benchmarks such as TurkCorpus and ASSET, we show that examples selected by the top SARI scores perform the best on larger models such as GPT-175B, while the compression ratio generally performs better on smaller models such as GPT-13B and GPT-6.7B. Furthermore, we demonstrate that MBL is generally robust to example orderings and out-of-domain test sets, and outperforms strong baselines and state-of-the-art finetuned language models. Finally, we show that the behaviour of large GPT models can be implicitly controlled by the chosen metric. Our research provides a new framework for selecting examples in ICL, and demonstrates its effectiveness in text simplification tasks, breaking new ground for more accurate and efficient NLG systems.
Involved external institutions
Turkey (TR)How to cite
APA:
Vadlamannati, S., & Şahin, G.G. (2023). Metric-Based In-context Learning: A Case Study in Text Simplification. In C. Maria Keet, Hung-Yi Lee, Sina Zarriess (Eds.), INLG 2023 - 16th International Natural Language Generation Conference, Proceedings of the Conference (pp. 253-268). Prague, CZE: Association for Computational Linguistics (ACL).
MLA:
Vadlamannati, Subha, and Gözde Gül Şahin. "Metric-Based In-context Learning: A Case Study in Text Simplification." Proceedings of the 16th International Natural Language Generation Conference, INLG 2023 - held jointly with the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDial 2023, Prague, CZE Ed. C. Maria Keet, Hung-Yi Lee, Sina Zarriess, Association for Computational Linguistics (ACL), 2023. 253-268.
BibTeX: Download