COMPASS: A Process Mining-based Methodology for Prompt Optimization of Large Language Model Agents
Dorsch R, Henselmann D, Harth A (2025)
Publication Type: Conference contribution
Publication year: 2025
Publisher: CEUR-WS
Book Volume: 3996
Pages Range: 29-37
Conference Proceedings Title: CEUR Workshop Proceedings
Event location: Vienna, AUT
Abstract
Exploring and optimizing the behavior of LLM agents in complex environments is challenging. We present COMPASS, a methodology that systematically applies Process Mining to discover, analyze, and refine agent behavior. COMPASS consists of five phases: planning the project, extracting data from the LLM agent, transforming data into event logs, exploring and analyzing agent behavior, and providing feedback through conformance checking or prompt design guidelines. We evaluate COMPASS in a case study with an LLM agent that uses three tools to generate SPARQL queries while exploring a complex knowledge graph. Through COMPASS, we identified ineffective behavior patterns and optimized prompts to improve performance. Our methodology supports data-driven prompt optimization with interpretable behavioral models, enhancing explainability and reliability in complex environments. This ongoing research aims to systematically improve agent performance through exploratory behavioral analysis that increases transparency in agents’ decision-making processes.
Authors with CRIS profile
Involved external institutions
Germany (DE)How to cite
APA:
Dorsch, R., Henselmann, D., & Harth, A. (2025). COMPASS: A Process Mining-based Methodology for Prompt Optimization of Large Language Model Agents. In Janis Grabis, Yves Wautelet, Emanuele Laurenzi, Hans-Friedrich Witschel, Peter Haase, Marco Montali, Cristina Cabanillas, Andrea Marrella, Manuel Resinas, Karolin Winter (Eds.), CEUR Workshop Proceedings (pp. 29-37). Vienna, AUT: CEUR-WS.
MLA:
Dorsch, Rene, Daniel Henselmann, and Andreas Harth. "COMPASS: A Process Mining-based Methodology for Prompt Optimization of Large Language Model Agents." Proceedings of the Joint of the 3rd International Workshop on Hybrid Artificial Intelligence and Enterprise Modelling for Intelligent Information Systems, HybridAIMS 2025 and the 1st Workshop on Compliance in the Era of Artificial Intelligence, CAI 2025, Vienna, AUT Ed. Janis Grabis, Yves Wautelet, Emanuele Laurenzi, Hans-Friedrich Witschel, Peter Haase, Marco Montali, Cristina Cabanillas, Andrea Marrella, Manuel Resinas, Karolin Winter, CEUR-WS, 2025. 29-37.
BibTeX: Download