TestProject was proud to join forces this year with Flood and SpecFlow to conduct the first ever State of Open Source Testing Survey. Over 1,800 members of the Tricentis community came together to share their opinions on open source testing best practices, tools, and trends.
Nearly 67% of the total respondents were involved in regular functional testing, making it the most commonly performed type of testing surveyed. While we weren’t surprised to know that functional testing plays a critical role in the software delivery process, we were interested to discover the following specific trends:
Finding #1: 77% of companies perform Functional Testing frequently, at least with every sprint
With the increase of CI/CD and DevOps, fast feedback is essential to ensuring developers know when hasty code commits lead to unexpected bugs. Testing with every sprint is imperative, and testing with every build is becoming the norm. This rapid frequency of testing demands automated functional testing which is not only reliable, but has substantial coverage to ensure no major bugs are being overlooked.
Finding #2: 84% of companies still expect QA and Testers to lead functional testing efforts
A rapid increase in the frequency of builds has resulted in many companies experimenting with developers writing functional tests for their own code. This way, there would be no bottleneck forcing developers to wait for testers to validate their new features before they could be released. However, only 8% of respondents have successfully shifted this duty to developers. When it comes to time, developers are already under pressure to deliver faster than ever before, so it is not surprising that they are unable to take on further responsibilities.
Finding #3: 92% of organizations want open source functional testing tools
While we knew that open source tools were gaining market share quickly over their commercial counterparts, we never could have expected to see that they were practically mandated across the industry. Part of the rise in popularity is likely due to the need for tools that can seamlessly integrate into CI/CD pipelines and processes, while some may also be due to the increasing scrutiny of tool budgets in the current economic climate.
Finding #4: Selenium and Appium still reign supreme, as do tools built to leverage them
Over 80% of respondents surveyed have experience with Selenium, making it the most popular open source functional testing tool surveyed. Appium wasn’t far behind, with 35% of respondents surveyed having experience with it, equally impressive due to the smaller community of mobile developers and testers. We were also impressed by the popularity of our own TestProject (19%) as well as Katalon (17%), which are both free tools built on top of Selenium and Appium.
Finding #5: Open source functional testing is not without its challenges, namely time to invest in it
The survey results clearly outline that open source tools have the potential to unlock amazing efficiencies in functional testing, however, they certainly have their challenges. More than 50% of survey respondents stated one of the biggest roadblock to adoption was finding enough time to get familiar with the tools. Not far behind that, 40% mentioned not having the technical skills needed to be proficient in the tools. With the current economic climate giving many people more time to invest in themselves and their skills, we are anxious to see if these challenges get lessened in the coming months.
Finding #6: People problems are just as likely to derail open source testing initiatives as the tools themselves
When given the chance to share their individual thoughts on open source testing through an open response, the overarching theme was not to overlook personnel and culture related challenges to implementing new tools. Providing adequate training, gaining the necessary team buy in, and navigating the change management landscape were all challenges that many respondents wished they had seen coming. Ultimately, it is clear even the best tools cannot overcome teams that are not ready and willing to adopt them.
All in all, we felt this survey confirmed that there is no more exciting time than the present to be involved in open source testing. Make sure to head over to the full infographic to see the detailed results and our additional findings on behavior driven development and load/performance testing.
We also invite you to register for a free webinar, May 27th, to hear us discuss all about these exciting survey findings – Save your seat today!
We look forward to updating our findings with another survey in 2021. In the meantime, enjoy all the great open source testing tools on offer, and happy testing! 💪🌟