Element Re-identification in Crowdtesting

Software usually provides different GUI layouts for different devices for a better user experience. This increases the workload of testing, so crowdsourced testing is needed to reduce costs. The crowdsourced testing will perform similar test steps for each GUI layout and record them. After the software is updated, each GUI layout can be automatically tested according to these test records. A test record contains several steps, and each step contains an operation and an element. The automated test is to find the element according to the recorded element attributes and then perform the recorded operation. The idea of manually testing one GUI layout and then automatically testing other GUI layouts does not work. Because an element may have different attributes in different GUI layouts, the attributes recorded in one GUI layout cannot guarantee that the elements will be found correctly in another GUI layout. However, humans can easily find the same element in different GUI layouts. This is because the appearance of the same element in different GUI layouts is similar. Humans can easily perceive this with their eyes, and so can AI. To achieve this, we propose an approach of visually re-identifying elements. Specifically, our method consists of two convolutional neural networks, Element Re-Identification Network (ERINet) and UNet. ERINet can identify whether two elements are the same or different. UNet provides ERINet with attention masks of elements and backgrounds which can help improve the accuracy. Furthermore, we introduce a new dataset for element re-identification, which contains 31,098 element images and 170 background images. Our method achieves excellent performance on this dataset. Our code and dataset are made publicly available at https://github.com/laridzhang/ERINet.