An Approach to Workload Generation for Cloud Benchmarking: a View from Alibaba Trace

Finding performance bottlenecks through bench-marking is one of the driving forces to improve the resource provision efficiency of cloud computing. Although existing benchmarks have been designed to improve the effectiveness in system performance evaluation, the following problems still exist in these benchmarks due to insufficient consideration of the characteristics of jobs in the production environment: (i) lacking of understanding for the details of workloads composition in the production environment, which reduces the authenticity of the job. (ii) the design of workloads submission patterns lacks quantization and reproducibility, which often relies on a random setting. In our benchmarking, multiple workloads are generated by analyzing and fine-grained matching the composition of workloads in the real production, and a design of workloads submission pattern based on LSTM time series prediction is proposed to simulate the real submission behavior. We finally demonstrate the effectiveness of our work by evaluating the impact of different workloads submission patterns on system performance evaluation.