The question arising at each stage of development is that regression after each iteration takes too long, and the functionality being verified often had no changes. Consequently, the possibility of defects is quite low. As a natural result, we arrive at an idea to automate all regression scenarios and refuse manual regression testing. The advantages here are obvious:
- automated tests are faster than a human;
- automated tests can be performed at any time;
- automated tests are very precise;
- automation can be used in almost all testing processes
- automatic creation of reports.
But unfortunately, regression tests automation has many pitfalls that are rarely discussed.
Before you start to automate regression testing is necessary to solve a few questions:
- What functionality is to be covered by the tests;
- Autotests architecture.
- Choice of a tool for regression test automation.
Let us pass on to the real situations in practice. At the stage of switching from manual testing to automated tests, I got a project where I had to test the registration of two types of accounts for the site on 43 domains depending on the country. It was just as hell. The client didn’t care about the quality of the information displayed. The requirements were as follows:
- accounts are created without any fatal errors;
- input information coincides with the values that are displayed in the settings after creation.
After a couple of runs I realized that I’ve had enough. At that moment I could only see that the number of registration fields is different for some countries, and I also had some coding experience in C#. Another regression was close at hand and I could not hesitate. I asked my friend familiar with automation to tell and show how to write tests. After many questions, tries and errors a simple navigation test was born. I discovered locators and Selenium web drivers – and lo and behold, all the domains used the same locators. I had a little left to do – to run the last manual regression in my life on all domains and fill the table with the fields and corresponding domains. Another round of manual regression – a long process with completing the table – and seems like is all over.
I enjoyed every other regression. I could watch some TV series while regression was running. By the way, it was just as long as one episode. While I used to spend about 4-5 hours on registration testing on 43 domains, automated regression testing took me 40-45 minutes, provided non-optimal code, no report system and no proper exception handlers.
Considering the above said we can make conclusion #1: to see the advantages of test automation, we should start with the most repeated tests, which are usually positive, but they need to be run after each release.
This continued for several releases until the company decided to automate most of the functionality officially. Since the client was not well-versed in automation tests we decided to automate any and all that is within my knowledge on test automation.
As a result, we have covered almost the entire system with autotests. All would be fine, but after any fixes some tests started to fail. Mostly, it happened because of long scripts that contain a lot of steps:
- Test could fail even at the second step, and as the first part of the test usually brings the system to a certain state, running the whole test had no sense;
- Long scenarios contain much code and actions like that; as a result, repeating the first step might lead to failure of many tests.
- Because of the code repetition, autotests check the same point for many times; this results in needless increasing of the test duration.
On that basis we can make conclusion #2: too long scenarios should be run manually, as a lot of time is spent on writing them and they can often fail because of some minor bugs or errors at the second step.
Let us consider another very interesting situation. The client ordered autotests as usual and asked to write a couple of examples. He approved the idea, and in several days we added test automation to the plan. But our programmers did not meet the deadline, eventually we had to write autotests based on the current stage. With a delay of a couple of days programmers had a release, but more than half of autotests failed.
Conclusion #3: never start writing automated test without a stable build.
Only a short time ago we had one more funny situation. We had received an order for site automating and have done everything quickly. Everything went well more than ever. A few weeks later the customer sent a letter complaining that tests did not work. As it turned out, he started a site redesign and all the tests had failed. It taught me to do stable tests.
Therefore, conclusion #4 is: when possible, use the locators so that the elements could be found after the slightest changes.
Therefore, the advantages of manual testing are seen clearly – it doesn’t depend on anything and can be done all the time. Abandoning manual testing will bring nothing good. Automated and manual testing are interrelated and complementary testing methods and each has pros and cos. Before thinking of regression scenarios automation consider time expenditures for both writing tests and their support. Also, pay attention that manual specialists are usually paid less than those who have skills on test automation.
So, as always, the whole issue is about money. If automation of regression testing scenarios saves money without losing the quality of the product, you can expand the range of automated scenarios until the product quality is high and you save costs. The way of balancing here is up to you to decide and depends on personal experience and preferences.