Toward GenAI-Based Contextual Metadata Generation for Reusability: Evaluating Metadata Extraction from Scientific Publications

Metadata is crucial to improve the findability and reusability of research outputs, thereby contributing to a more efficient use of research resources. Their effectiveness increases as metadata goes beyond purely descriptive elements and captures richer contextual information. However, creating such context-rich metadata remains a time-consuming task for researchers. Recent advances in Large Language Models (LLMs) offer the potential to provide scalable, automated metadata annotation to support research communities. Therefore, we pursue the development of an LLM-based metadata annotation service that automatically generates structured metadata, aimed at reducing documentation effort for researchers while improving findability and reusability. The deployment of such a service, however, raises critical quality concerns. Manual metadata annotations are commonly treated as ground truth, despite being selective, potentially incomplete, and influenced by interpretation. Conversely, LLM-generated metadata may introduce additional uncertainties, including hallucinated metadata elements. To address this challenge, we conduct a comparative evaluation of metadata annotations generated by human annotators and one LLM (ChatGPT-5.2) from 22 scientific publications using an identical structured schema. Annotation quality is assessed along two complementary dimensions: completeness, capturing coverage of publication-supported metadata elements, and semantic correctness, measuring the degree to which annotated metadata values are textually grounded. Our analysis indicates that LLM-based annotations achieve higher completeness across most metadata categories, particularly for contextual and process-oriented elements that are frequently omitted in manual annotations. Importantly, although the higher completeness of LLM-based annotations might intuitively suggest lower semantic correctness and an increased risk of hallucinated metadata elements, our experimental results do not support this assumption. By operationalizing metadata quality through explicit dimensions, this work provides measurable criteria for evaluating automated metadata annotations.