The “Internet of Things”, or IoT as it is popularly called, has taken the world by storm. Everywhere around us, we find ourselves surrounded by IoT devices or sensors. From smart wearables, smart cars, smart manufacturing, smart farming, smart shopping, and smart pharmaceuticals to smart cities, it’s IoT all around us.
While we are having an experience like never before and ease of life, it has also given a huge headache to the developers and testers all around the world 👩💻 Finding new strategies and developing code that would help run this huge ecosystem is a big challenge. As IoT is a mesh of hardware + software + firmware + protocols + operating systems + network, developing test strategies and scenarios is a challenge that wasn’t faced earlier.
What is the challenge?
There are a few basic challenges:
💢 IoT is neither software nor hardware. It’s a combination of both that needs to be tested simultaneously. While testers are used to testing either of these, modern-day testers need to test both. They need to perform a variety of tests, ranging from functional tests to network bandwidth tests. Basically, they need to test the whole IoT ecosystem which is a big challenge.
💢 To test, we are used to developing a test lab with the software under test (SUT) installed. That gives us a real-world prototype to be tested. With IoT, developing a virtual test lab is not easy. Real-life scenarios are highly scaled-up versions of the in-lab tests, hence the lab results cannot be relied on fully.
💢 The testing process itself needs a change. The data used to come as user input, but now the data comes in from a variety of user devices like accelerometers, gyroscopes, light sensors, altimeters, GPS locators, and myriad other types of sensors. To assess and finalize the data structures is a huge challenge in itself.
💢 The IoT devices handshake with each other using various protocols like XMPP and MQTT. All the protocols should be compatible with each other in order for the data to flow smoothly. With such amounts of data, data security, and privacy, it becomes a huge issue and testers should find strategies to test these.
As mobile testers have discovered, smart device testers will have to test their devices in the field. Imagine real-world scenarios to test outside of the lab. We will encounter loose network connectivity, location detection can be imperfect, and step sizes can vary between short decorative steps in a parkway to the terraced rock, leading to Mount Everest 🗻
In addition to vetting the sensors in the smart device, testers will gain insight into the UX. For example, do the numbers display large enough and bright enough to read when you’re jogging at night? If yes, will it be this shiny when the user is wearing the device and sleeping?
How should I approach testing IoT devices?
There are thousands of testing types and many of them will be used here. We should know beforehand the industry we are testing for and pick some industry-specific tests. Add to it a generalized testing set and your initial test plan is done. Take a look at this image which will show the tests to be carried out at each layer.
1️⃣ Application layer
More intelligence is extracted at this layer by processing and analyzing data. IoT systems are linked to middleware or software that can better comprehend data. The following testing types need to be carried out here:
2️⃣ Services layer
This is the layer at which inter-operability takes place. The following tests need to be carried out here:
- Inter-operability between protocols and devices
- API testing
- Functionality testing
3️⃣ Gateway and network layer
This is where the exchange of network packets takes place. The following tests need to be carried out here:
- Network connectivity
- Network compatibility
4️⃣ Sensor layer
These IoT layers are part of the internet’s physical architecture, serving as a link between the digital and physical worlds. The following tests need to be carried out here:
- Functionality testing
- Security testing
🔎 Here are some examples of tests that need to be planned before you begin your test setup and actual testing:
- Performance– Needs to be done at the Network and Gateway level (protocols like MQTT, CoAP, HTTP), System level (Database, processing, analytics), and the Application level. E.g.: Verify response time against bench-marked time with defined connectivity conditions.
- Security– Sensor networks, real-time data collection applications, middleware, M2M protocols, and interfaces are just a few variables that could bring in more injectable points and new security threats. e.g.: Verify no unauthorized access to the device or verify the data on compromised devices can be remotely wiped out.
- Compatibility– A major necessity in the Application layer and the Network layer of the IoT framework. It’s all about ensuring that the device hardware, communication protocol versions, software versions, and operating systems are all compatible. e.g.: Verify IoT software supports a defined set of devices or ensures device-to-device communications protocols are compatible.
- End-to-End application testing– Includes the testing of all functional use cases of an IoT application including user experience and usability testing. e.g.: Verify IoT application has all required features working as per the specifications or verify whether the User Experience (UX) is good.
- Device interoperability– It verifies the connectivity across all the devices and protocols in the IoT setup. Because IoT requires standards to allow platforms that are communicable, operable, and programmable across devices, interoperability testing at the Service layer of the IoT architecture becomes extremely important, regardless of make, model, manufacturer, or industry.
Conclusion
The IoT testing approach is different and should be derived from the system and architecture involved. The old way of testing may not be 100% applied here, so think outside the box and develop new strategies.
This is more of a TaaS (testing as a user) approach rather than a requirements based one, hence it’s focused more on field testing. It’s challenging but exciting, as testers need to test and certify a complete mesh of devices, protocols, hardware, software, firmware, and operating systems 🤩