With the rapid development of the mobile Internet and smart devices, mobile applications are becoming increasingly complex in both functionality and user-interface interactions. Ensuring application quality has therefore emerged as a major challenge in software engineering. As graphical user interfaces (GUIs) serve as the primary medium through which users interact with mobile applications, GUI testing plays a critical role in mobile application quality assurance.

Traditional manual GUI testing can identify usability issues, interaction anomalies, and interface defects. However, it relies heavily on testers’ experience, is time-consuming, and struggles to keep pace with the frequent iterations and continuous delivery of modern mobile applications. Existing automated GUI testing techniques typically explore application interfaces using random, model-based, or reinforcement-learning-based strategies to improve testing efficiency and reproducibility. Most of these approaches, however, optimize primarily for code coverage or state coverage, with limited understanding of application business logic and real-world user workflows. As a result, they often generate test sequences that lack semantic coherence and fail to cover critical yet complex functional scenarios.

In manual testing practice, testers usually organize test cases around specific scenarios, such as sending an email, completing a purchase, checking the weather, or logging into an account. Such scenarios represent more than a sequence of interface actions: they capture the relationships among functional goals, business logic, and user intent. Based on this observation, the iSE Lab, in collaboration with the Chair of Software Engineering and AI at the Technical University of Munich, has proposed ScenGen, a large language model-based framework for scenario-guided GUI testing of mobile applications. By leveraging the ability of large language models (LLMs) to understand interface semantics and testing intent, ScenGen automatically transforms high-level test scenarios expressed in natural language into executable, reproducible, and traceable GUI test sequences, thereby bridging the gap between low-level interface interactions and high-level business logic.

Specifically, ScenGen employs a multi-agent collaboration mechanism to emulate the manual testing process, dividing scenario-guided GUI testing into five stages: observation, decision-making, execution, supervision, and recording. The Observer perceives the current application interface and combines conventional computer vision techniques, optical character recognition, and multimodal LLMs to identify interface components, layout structures, and semantic information. Based on the target test scenario, current GUI state, and interaction history, the Decider determines the next target component and the action to be performed. The Executor translates the decision into low-level executable commands and performs actions such as tapping, text entry, swiping, and navigating back on a mobile device. The Supervisor validates the execution results in real time, determining whether the application has finished loading, whether the state transition matches expectations, and whether the target scenario has been completed. The Recorder captures interface states, action sequences, runtime logs, and execution outcomes, organizing them into reusable contextual memory and test scripts. Through this collaborative mechanism, ScenGen maintains a continuous understanding of the scenario goal throughout the testing process, preventing the test from deviating from the intended business workflow or becoming trapped in irrelevant exploration.

To improve the stability and executability of scenario-guided testing, ScenGen further introduces automated component localization and self-correction mechanisms. Although multimodal LLMs possess strong semantic understanding capabilities, they often struggle to provide precise pixel-level interaction coordinates. ScenGen therefore separates semantic decision-making from component localization. An LLM first determines the intended action and the semantic identity of the target component; the target is then matched, predicted, and spatially adjusted using visual recognition results. When recognition results are incomplete or the interface layout is complex, ScenGen predicts potential target locations based on neighboring components and spatial relationships. For components that require precise interaction, such as text fields, ScenGen automatically adjusts the tapping coordinates so that the action falls within the actual interactive region. If an action fails to produce the expected state change, the system activates its self-correction mechanism, reanalyzes the cause of the failure, and generates a revised action, thereby improving the robustness of the testing process.

To address the limited understanding of business logic, weak semantic coherence of test sequences, and insufficient adaptability to dynamic interfaces in existing automated GUI testing approaches, this study presents an LLM-driven, multi-agent GUI testing framework centered on test scenarios. Unlike traditional coverage-oriented automated exploration, ScenGen focuses on real-world user goals and business-critical workflows. It generates reproducible test scripts through the coordinated use of semantic perception, scenario-based decision-making, execution validation, and contextual memory. This work further expands the application of LLMs in software testing and provides a new technical path toward intelligent GUI testing systems that are business-aware, explainable, and reusable.

The research paper, titled "Scenario-Guided LLM-based Mobile App GUI Testing," has been published in ACM Transactions on Software Engineering and Methodology, a leading software engineering journal recognized as a CCF-A venue.

The paper’s first author, Dr. Shengcheng Yu, received his doctoral degree from the School of Software at Nanjing University in 2024 and is currently a postdoctoral researcher at the Technical University of Munich. The research was jointly supervised by Professor Zhenyu Chen and Associate Professor Chunrong Fang of Nanjing University, and Professor Chunyang Chen of the Technical University of Munich. Dr. Yu’s research focuses on intelligent software testing, automated GUI testing, and crowdsourced testing. He has conducted a series of studies on software quality assurance based on knowledge understanding, scenario modeling, and large language models. Building upon earlier work on scenario-based GUI testing, this study further incorporates LLMs and multi-agent collaboration, offering an important reference for advancing automated mobile application testing from interface exploration toward business-level understanding.

Paper preprint: https://arxiv.org/abs/2506.05079

Project repository: https://github.com/iGUITest/ScenGen