Along with Continuous Delivery and Continuous Integration, Continuous Testing enters the game. The notion started spreading a good five years ago, yet today Continuous Testing is not ubiquitous. Why so? Read our article to find more about a not-so-easy transformation of the testing universe.

A Start of Continuous Testing Era

Why all the fuss about Continuous Testing? How come that testing procedures, usually left almost forgotten up to the end of the SDLC in the past, are now required at every project stage and have finally acquired the same level of significance as development in terms of project delivery. Historically, test teams were given far less time and resources than dev teams, as testing was considered sort of a supporting service and responsibility for the project outcome was mostly laid on the genius of a developer.

See the time/significance interdependency of the SDLC phases in the picture below.

Picture 1

However, the things have changed, to a great degree due to the transformation of industry appetites. Today’s releases are much more frequent than they used to be, say, five years ago, and the price of a mistake is times higher now. Consequently, an app that is released monthly, weekly, daily or more frequently, demand a different approach to code writing and testing as compared to the traditional waterfall-based cycles, and will be created under the laws of Agile and DevOps. It came only natural that extreme appetites of the IT-industry spurred a new-level, extreme development and testing.

Another contributing factor to the rise of a new testing era was, actually, the rise and fall of test automation. Test automation, when applied appropriately, gave us a speedy delivery, costs reduction and eliminated human factor from our testing procedures. Nevertheless, since test automation isn’t an option for many projects and many companies still have to do the job manually (see why here), global IT-community had to admit that Something went wrong with such a brilliant thing as test automation and It didn’t work as intended. It failed indeed, and 70% of global testing which is still done manually confirms that.

Continuous Testing Steps in...

At this sad moment, Continuous Testing steps in and joins Continuous Delivery, which, frankly speaking, came as no surprise as “Continuous Delivery Needs Continuous Testing”, according to Forrester.

Continuous Testing was the bullet to hit the two mentioned above objectives – speedy and high-quality delivery according to the best Agile practices and rehabilitation of test automation. Let’s study the notion in more detail.

Continuous Testing was designed to enable uncovering of defects and bugs at the earliest stage possible through “atomic” tests as Adam Auerbach, Vice President of Quality Engineering at EPAM Systems, calls them in his article. These small, independent units, which have no dependencies with other tests, are launched each time a change is introduced to the system. In such a way CT ensures quick and reliable feedback on the current state of the system, i.e. tells if the change has negatively affected the system, and gives go or no-go results to the developer.

Key items of continuous testing are PEOPLE, PROCESSES and TECHNOLOGIES. As for people, Continuous Testing implies empowering developers with testing capabilities. CT philosophy urges the developers and agile teams to perceive testing as part of their responsibility. Processes within CT vision are characterized by ultimate automation. As regards the technologies, these should be cutting-edge open-source ones which support automation and are easy-to-use for the developers.

However, as it goes, these three notions are also the key impediments to the introduction of continuous testing at a global scale. Why? See below.

Key items = Key Impediments to Continuous Testing

PEOPLE

What a company leader should undertake to make its people not simply sit at the office desk 9 to 5 but continuously add value to the processes they are involved in? This is not an easy nut to crack, however this issue, if resolved, will help a company apply CT philosophy in its daily activities. People are very reluctant to wear many hats, though CT demands exactly this. “The people involved, developers and testers, must become part of this new culture. There should be a DevOps organization set up to hire or retrain people into more well-rounded engineers who can wear many hats”, - The Definite Guide to Continuous Testing says. Developers capable of worthy testing and testers enabled to write a healthy code, that’s what this new culture requires. Here's the tricky bit: how many companies are truly adequately staffed for this transformation?

Moreover, attitude towards automation, which lies in the core of CT, can also be a challenging aspect. As automation has proven to be quite an ambiguous benefit from time to time (due to its high cost and numerous application limitations), management along with the staff may consider manual, not automated tests to be a safer option for the project. This opinion is reasoned in Impediments for software test automation: A systematic literature review. In this paper, the authors have studied, among others, behavioristic impediments to test automation application. See the abstract below accompanied by the picture for better comprehension.

“Consider the case where automation is not viewed as useful. This will likely reduce the willingness to invest in the automation by allocating people and time to the work. Less time and people available to do the work will likely lead to a low quality automation. Low‐quality automation is not as useful as high‐quality automation, which reduces the perceived value and the feedback loop is closed”.

Picture 2

That being said, CT, which actually is, to a great deal, about multitalented and super involved people, can stall right at its start. For this not to happen, consider continuous talent training programs and ensure that well-tuned corporate processes and cutting-edge technologies accomplish the appetite for perfection and value of your talents.

PROCESSES

As a risk-based alternative to the traditional all-encompassing testing practices, the processes in CT should be tuned so as to quickly unveil fresh bugs after adding new code to the system through targeted automated tests. Fail fast is the mantra of the followers of the CT-philosophy. Why worry about such a trifle thing as small bug, if a 100% testing is impossible anyway? The thing is, such bugs, can potentially cause a specific risk to materialize, which a company identifies as significant for the app in question. That’s why the processes should be corrected accordingly to enable targeted automation testing of small code portions.

Got it, what’s wrong here? As it happens, automation has a limited application scope. Under the lift-shift testing rules which apply to CT, “developers test as they go, with test automation including functional testing, performance testing, monitoring and security testing” (DevOps). Among the above mentioned test types only two – functional and performance testing – can be automated with sufficient degree of reliability.

As Don Prather, Technical Program Manager at Axway Software, states in his article, tests launched during Continuous Testing should meet flexibility, coverage and effectiveness requirements. In other words, tests must be flexible so that they can be run at any time; test coverage should be maximum within a given period to test; effectiveness of the tests must ensure immediate detection of hard-to-pinpoint defects. Thus, functional and performance tests are perfect candidates for CT-implementation due to their nature, as they are almost never done manually and meet the above requirements.

However, as for security tests automation, this area is still in its infancy. Justin Arbuckle, vice president of worldwide transformation and chief enterprise architect at Chef, highlights that many enterprises hold onto manual security testing because of their concern with verifiable and auditable trails. “Companies want to have someone reviewing every line of code to make sure it’s up to regulation standards and there aren’t any holes leaving a company open to breaches”. Hence, a huge lump of testing work is still done manually, not to mention UI/UX testing, Exploratory testing and even sometimes API testing (though API test automation is a must for CT-implementation).

TECHNOLOGIES

For the developers to be able to test out their work, developer-friendly tools are required. In terms of availability, this is not a problem anymore, for today one can find various automation tools for many application purposes (security test automation, API testing tools, GUI testing tools and many others). However, the cost of such technologies can be devastating to the efficiency of the project, and CT, as well as CI and CD are all about achieving the highest efficiency possible.

Do Nguyen (LogiGear) supports the idea and includes start-up costs to the major barriers to Continuous Testing. “The initial effort required to pick, setup and configure the tools and frameworks to get to a running automation process can be daunting. The heavy investment for research time and substantial financial investment coupled with the added possibly of interrupting your day-to-day operations have been a big deterrent for many organizations”.

Is there a chance to do without top-notch testing tools in achieving CT-objectives? The Definite Guide to Continuous Testing says No. The fact is, many companies stating they stick to the CT-approach in their projects, continue working with “legacy” tools which sufficed for waterfall. “They [legacy tools] are an impediment to testing at the speed of agile. They don’t support automation in the right way for a continuous integration environment and they are tools that developers refuse to use. There needs to be a new set of tools that won’t get in the way of speed and quality; tools that enable quality to be built into the process and not “just” test the application as in the past”.

Conclusion

As we see, revolution in the testing industry cannot run as smoothly as planned. However, much was, is and will be done to enable a full-scale transformation of the sector. CT promises a lot: speed, quality, and new revenues, and in spite of many initial blockers, it will find its way to the acknowledgement by the global IT-community, like Agile, DevOps, CI and CD did.

For more information about our Test Automation practices, contact our Support team or visit our website. Qulix Systems is renown for its QA and Test Automation services and pays immense attention to the professional advancement of its testing engineers.