Causes and effects of fitness landscapes in system test generation: a replication study

Search-Based Software Testing (SBST) has seen several success stories in academia and industry. The effectiveness of a search algorithm at solving a software engineering problem strongly depends on how such algorithm can navigate the fitness landscape of the addressed problem. The fitness landscape depends on the used fitness function. Understanding the properties of a fitness landscape can help to provide insight on how a search algorithm behaves on it. Such insight can provide valuable information to researchers to being able to design novel, more effective search algorithms and fitness functions tailored for a specific problem. Due to its importance, few fitness landscape analyses have been carried out in the scientific literature of SBST. However, those have been focusing on the problem of unit test generation, e.g., with state-of-the-art tools such as EvoSuite. In this paper, we replicate one such existing study. However, in our work we focus on system test generation, with the state-of-the-art tool EvoMaster. Based on an empirical study involving the testing of 23 web services, this enables us to provide valuable insight into this important testing domain of practical industrial relevance. Our results indicate that fitness landscapes are largely dominated by neutral regions (e.g., plateaus), which make the search process challenging. We observe that the presence of information content in the landscape can improve search guidance, while boolean flags are a primary contributor to neutrality. These findings confirm prior results in unit testing but also reveal system-level differences, particularly in how branch types impact search effectiveness. These insights suggest the need for improved fitness functions, testability transformations, and search operators tailored to system-level testing.