Test Automation Advantages and Disadvantages
Test Automation, when done correctly can have many advantages and be very beneficial to the project and organization. There are however some pitfalls or disadvantages of test automation that we need to be aware of.
Test Automation Advantages
What are the advantages of Test Automation?
Confirmation of the known
Automated checks are a great way of confirming that the application still functions properly after changes made to it.
It is possible that when a new feature is added to an application or a bug is fixed, it impacts the functionality of the working software, i.e. a regression bug is introduced.
By running a set of automated regression checks when the application is updated, we can identify any new bugs introduced as a result of the changes.
The key information here is to run automated checks as often as the application is upgraded.
No need to run the full set of automated checks. A quick smoke regression pack should be enough to pick up any major issues.
Quick feedback
Another big advantage of automated checks is the quick feedback we get when the application is updated. Ideally the development team should fix any failures as soon as they arise, before moving on to other tasks.
Please note that this quick feedback can only be achieved with unit tests and API tests. If we test functionality from the UI or at system level, tests can take a long time to complete.
Fast execution of checks
Automated checks can take a while to script. However, when we execute them they are generally fast and can go through various steps much quicker than a human. Therefore, they help with providing a quick feedback to the development team.
This is especially true in the case of data driven scenarios.
Frees up the time of the testers
The best use of automated checks are the regression tests.
Automating regression tests frees us the time of testers, so they can focus more on exploratory testing of new features.
By the same token, when implemented properly, automated checks can run automatically with minimum or no supervision or manual intervention.
The development team can contribute
Automated checks are usually written in the same language as the application under test. For this reason, the responsibility of writing, maintaining and executing the tests becomes a shared responsibility.
Everyone in the development team can contribute, not only testers.
Test Automation Disadvantages
What are the disadvantages of Test Automation?
False sense of quality
Beware of passing tests! This is specifically important in verifying functionality at UI or System level.
An automated check only checks what is been programmed to check.
All automated checks in a test suite can happily pass, but there could be major flaws undetected. The reason for this is because the automated check were not coded to “look” for those failures.
Solution: ensure you design good test scenarios before automating them. An automated check is only as good as the design of the test. Also complement automated checks with manual / exploratory testing.
Not reliable
Automated checks can fail due to many factors. If automated checks keep failing due to issues other than genuine bugs, they can raise false alarms.
For example, automated checks can break because of a UI change, a service being down or problems with the network.
These issues are not directly from the application under test but could impact the result of the automated checks.
Solution: Where possible / applicable, use stubs. Stubs overcome issues with connectivity or changes in the 3rd party systems. Therefore, the automated checks would be independent of any downstream failures.
Test Automation is not testing
Unfortunately, many people mistake “Test Automation” with Testing.
Once they have the tools to automate the testing, they want to “automate all the tests”. They want to get rid of all “manual testers”.
The truth is that testing is an exploration exercise. Testing requires domain knowledge, a focused mind and willingness to learn the application.
Testing is not just executing a set of pre-defined test steps and comparing the actual results with expected results. This is the job of automated checks.
To properly test an application, a human intelligence is always required.
Solution: Understand that for a successful delivery of a project you require both automated and manual testing.
One is not a replacement for the other; complement automated checks with manual / exploratory testing.
Maintenance Time and Effort
You have to accept the fact that automated tests require maintenance. As the application under test evolves, so should the automated checks.
Automated checks are short lived. If the regression packs are not kept up to date, you start seeing all kinds of failures.
Maybe some checks are no longer relevant. Or maybe the checks are not a true representation of the new implementations.
These failures could pollute the test results.
Embarking on test automation is not a one-off effort. To get the most out of the automated checks, they have to be kept up to date and relevant. This, requires a lot of time, effort and resource.
Solution: Because the maintenance factor is an ongoing activity, invest time in designing a good framework. Use reusable modules, separate the tests from the framework and use good design principles to alleviate the maintenance effort.
Slow feedback
When a functionality is ready to test, most of the time it is quicker to do a manual check.
The problem is, automated checks can take a long time to script depending on the complexity of the test. Therefore, doing a manual check gives quicker feedback than scripting, running and checking the results.
Also, in terms of UI and System level testing, automated checks can take a long time to complete and report. Therefore, if there is a genuine bug, we may not be aware until all tests have finished.
Solution: Try to automate the tests alongside development so that when development is complete, you can run the automated tests against the new functionality.
Also, separate the automated checks in different packs.
A smoke regression pack should be super quick. The tests should only check that the application can be started and accessed.
Next, you can have a functional regression pack which check major functionalities.
Another regression pack can include all the end-to-end tests and in-depth tests. These checks can be run overnight.
An example of a nightly run is the cross-browser automated checks. These typically take a long time to run on all browsers.
Not many bugs found
Majority of the bugs seem to be found by “accident” or when performing exploratory testing.
This is probably because in each exploratory testing session, we could be testing the application in different ways.
Automated regression checks on the other hand, always follow a given path. Sometimes with the same set of test data. This reduces the chance of finding new defects in the application.
Also the number of regression bugs seem to be lower than new-feature bugs.
Solution: Try to build randomness in the scenario and data. Trying different paths with different data each time can reveal potential issues.
Conclusion
In this post, we looked at some of the advantages and disadvantages of automated testing. When we are involved in automating tests, we should consider the above points to get the most benefit.