With increasingly more people becoming acute in automation, the natural need is to improve test automation quality and make tests more robust.
This article aims to provide a set of techniques to use in cases when you find your tests running but failing occasionally, then running for some time and failing for longer again.
Here are 7 steps you can take in such situations.
- Debugging and retries
If a test failed already three times, have it retry itself. This can increase the run times of the test, but it is just a quick fix until you take time to debug that cause properly. Thus, do this until a degree of test stability is achieved.
- Retrospective fixes
Under continuous integration setup, you have to consider if the executed tests actually should fail the build stopping the integration. When you’re over 30% test coverage, there are likely some tests not considered that important to cause the stop. At this point, investigate after your build goes out to retrospectively fix the problem.
- Intuitive stack-trace messaging
While creating page object models with descriptions like “search box” and “homepage”, it’s useful to ensure the descriptions are able to chain together to then throw in some natural language in order to make your stack-trace messaging more helpful for other testers to debug. This will be very handy for automation game beginners, not familiar with your tests, and those who simply find stack traces too baffling.
- Custom exceptions
Mind that your exceptions’ names are revealing enough. Creating a class which extends from an exception class allows you to make up your own custom exceptions and give yourself or other person running your tests a more information for debugging them.
- Pre-test checks
Flaky tests aren’t always the problem themselves, but the intermittently unready environment often is. Creating a set of tests to serve as a kind of “pre-flight” check is a possible way out. This test suite is to be run before integration tests in order to check all the moving parts your tests depend on are okay. This check should be very quick since it needs to check only HTTP statuses, and your integration tests will only run if these tests have passed.
- Tests in isolation
The failure of individual components testing in end-to-end tests can make the whole suite fail. In this situation a good idea is to write a smaller test that checks this failing component separately before the running of end-to-end tests.
Finally, it’s suggested to tag the tests so that they specify their run frequency. Also mind that each test can be added multiple tags.
While this is not an exhaustive list of possible solutions, the above techniques are a first-aid kit when your automated tests lack robustness.