Fuzzing microservices: A series of user studies in industry on industrial systems with EvoMaster

With several microservice architectures comprising thousands of web services in total, used to serve 630 million customers, companies like Meituan face several challenges in the verification and validation of their software. The use of automated techniques, especially advanced AI-based ones, could bring significant benefits here. EVOMASTER is an open-source test case generation tool for web services, that exploits the latest advances in the field of Search-Based Software Testing research. This paper reports on our experience of integrating the EVOMASTER tool in the testing processes at Meituan over almost 2 years (i.e., between October 2021 and July 2023). Two user studies were carried out in 2021 (with two industrial APIs) and in 2023 (with three industrial APIs) to evaluate two versions of EVOMASTER (i.e., v1.3.0 and v1.6.1), respectively, in tackling the test generation for industrial web services which are parts of a large e-commerce microservice system. The two user studies involve in total 321,131 lines of code from these five APIs and 27 industrial participants at Meituan. Questionnaires and interviews were carried out in both user studies with the engineers and managers at Meituan. The two user studies demonstrate clear advantages of EVOMASTER (in terms of code coverage and fault detection) and the urgent need to have such a fuzzer in industrial microservices testing. Given its clear advantages, EVOMASTER now has been integrated into the industrial testing pipelines at Meituan. To study how these results could generalize, a follow up user study was done in 2024 (with EVOMASTER v2.0.0) with five engineers in the five different companies. Our results show that, besides their clear usefulness, there are still many critical challenges that the research community needs to investigate to improve performance further.