To What Extent is Stress Testing of Android TV Applications Automated in Industrial Environments?

An Android-based smart television (TV) must reliably run its applications in an embedded program environment under diverse hardware resource conditions. Owing to the diverse hardware components used to build numerous TV models, TV simulators are usually not sufficiently high in fidelity to simulate various TV models and thus are only regarded as unreliable alternatives when stress testing such applications. Therefore, even though stress testing on real TV sets is tedious, it is the de facto approach to ensure the reliability of these applications in the industry. In this paper, we study to what extent stress testing of smart TV applications can be fully automated in the industrial environments. To the best of our knowledge, no previous work has addressed this important question. We summarize the findings collected from ten industrial test engineers who have tested 20 such TV applications in a real production environment. Our study shows that the industry required test automation supports on high-level GUI object controls and status checking, setup of resource conditions, and the interplay between the two. With such supports, 87% of the industrial test specifications of one TV model can be fully automated, and 71.4% of them were found to be fully reusable to test a subsequent TV model with major upgrades of hardware, operating system, and application. It represents a significant improvement with margins of 28% and 38%, respectively, compared with stress testing without such supports.