Internet of Things (IoT) ecosystems are pivotal in shaping user experiences, influencing home interactions, health monitoring, and automating daily routines. Nonetheless, the intricate task of designing, implementing, and testing software for these ecosystems poses challenges due to their diverse composition. These ecosystems typically consist of software operating on various platforms (e.g., mobile, cloud, embedded systems) tightly integrated through diverse network protocols. The complexity arising from this integration and the interplay of different software systems complicates software engineering processes.
Amidst the privacy and security concerns associated with IoT software, testing and analysis assume heightened significance. Ensuring both the functional and security/safety-related aspects of systems is imperative. Despite the critical nature of these testing activities, there is a dearth of knowledge regarding the current state of practices and the challenges faced by developers in executing them.
In this paper, we undertake a comprehensive investigation to unravel the landscape of activities and processes related to testing IoT ecosystems. Our approach involves a two-pronged study. Firstly, we mine and empirically analyze two prominent open-source IoT ecosystems, OpenHab and Home Assistant. Specifically, we scrutinize all testing-related code through rigorous open-coding analysis to delineate its various testing purposes.
Secondly, we conduct a survey involving 80 open-source developers engaged in IoT ecosystems. The survey delves into current testing practices, preferences, challenges, and expectations.
The culmination of qualitative and quantitative studies reveals ten key findings that illuminate testing practices, perceptions, and challenges encountered by developers in the realm of IoT ecosystems. These findings provide valuable insights for the research community, offering guidance in navigating the intricate landscape of testing practices for IoT software.