Many testing leaders over the last several years have been advocating shifting left, meaning getting involved earlier in the design and development process. By understanding the design and implementation of the application, testers can go into active testing with a more comprehensive understanding of its strengths and weaknesses.
The contrarian that I am, over that same period of time I’ve been advocating not shifting left, but shifting right. That is, continuing to test and monitor applications even after they were deployed.
Cloud deployment has made this both possible and necessary. While the cloud enables global reach and almost infinite scalability, it is far more complex than deploying to an enterprise data center. In most cases, testers generally run their tests on local systems or on simplified cloud systems that don’t well represent the deployment environment. If there are differences between the test bed and the production environment, testing can’t accurately reflect what issues might arise in production.
That’s why testers have to be responsible for monitoring and testing applications in production. The work isn’t done yet.
Convincing veteran testers that it is important to test in production can be a challenge. Despite acknowledging that the production environment may be substantially different than the test bed, testers have long-held biases against post-deployment testing. Tinkering with an application in production in the data center can have serious consequences, and testers are conditioned to think that their job is done once the application is turned over to IT.
But that’s simply not the case anymore. Recovering from an application failure can be faster in the cloud. At best, monitoring the application in production will likely speed diagnosis, as teams examine performance and availability data. At worst, it should be possible to roll back to a previous version if necessary, taking perhaps only minutes.
Also, monitoring and testing in the cloud enable testers to find issues before they become problems for users. It’s obvious why this is important in this era of digital user experience. Finding and fixing a bug before users encounter it is a significant win for the enterprise, because it doesn’t expose users to a bad experience, and perhaps lost productivity and sales.
Synthetic Testing in Production
Testers are familiar with automating user interface testing – that is, recording user steps and recording them for automated replay. In production, this is referred to as synthetic testing. It is often a part of a monitoring solution, so that the application can be monitored during testing.
These are tests that work through common user scenarios, often defined by the user stories that form the basis of the application. Once recorded, they can be executed at any time of the day or night. In doing so, they can test conditions of the application at different times and loads. You can even direct requests to different data centers in different parts of the world to better understand availability and response times.
This is also the best way of finding errors and other issues before users do. Synthetic testing can expose faulty logic or bugs that weren’t discovered during pre-production testing, even if users haven’t reported any problems yet. These issues can then be diagnosed and fixed before they impact any users or business.
Monitoring for Health and Efficiency
I haven’t yet said a lot about what we should be monitoring on a production application. First, we monitor some of the same things we do in the data center – CPU, disk, and network. Those remain fundamental in understanding how your application is behaving.
But with the increasing emphasis on user experience, the traditional monitoring metrics are taking a backseat to measures that affect real users. Some monitoring tools employ real user monitoring, or RUM, in order to check on time to return data, time to paint a new screen, and likelihood of getting a 404 or other error. This enables testers to report on deficiencies in user experience.
But the analytics associated with monitoring does more than give you a snapshot of application health. They provide information on trends, which enable testers to identify if an application may have a problem in the future. By storing large amounts of monitoring data over a period of weeks or even months, it’s possible to identify memory leaks, bottlenecks, or other issues before they cause larger problems.
Testers Need Monitoring Data
But testing also has to include continuous monitoring and testing in production for cloud applications. Why? Because testing has changed over the last several years. With practices such as DevOps and continuous deployment, as well as less of a distinction between phases of the application lifecycle, including production. In practices such as agile and DevOps, testers certainly have to shift right to make sure the application works as expected in production.
I’ve talked a bit about what to monitor and why, but not about how to monitor. There are a wide variety of application monitoring tools available. I’ll name a few here, but do your own research based on your unique needs. The most comprehensive commercial tools include Dynatrace, New Relic, AppDynamics, DataDog, and Sumo Logic, but there are literally dozens of tools to choose from.
There are also a variety of open source monitoring solutions. Geekflare has a description of six popular open source tools, while OpenAPM incorporates a flow that enables you to select the best solutions based on your stated needs.
I’m not advocating for any particular tool. Monitoring systems have different capabilities and price points, and there is no “one product fits all”. But monitoring for availability, performance, load, and errors has become critical to the quality of a cloud-based application. And the many solutions available attest to the notion of the importance of these activities.