What is the Typical Structure of an Automated Test?
How should we structure our automated tests? More importantly, how can we make our automated tests deterministic and repeatable?
Every automated test should be validating against one or more prescribed conditions. That is, while creating an automated test, we have to know what the outcome of an action should be so that we can put appropriate evaluation statements in place.
Automated tests should be run automatically with no manual intervention. Moreover, the results of the automated tests should be deterministic and reliable.
But how do we achieve this?
The answer is quite simple but one which often gets neglected and that is, the use of controlled data.
Components of a Test
To run a test, we need three components:
- Application
- Scenario
- Data
The one component that heavily influences the behavior of an application is data; Data that the application needs and the data that is fed to it.
To illustrate this, think about an e-commerce application where users can search for products.
The search results page would look and behave differently when there are products in the database than when the database is empty.
Obviously our automated tests should cater for different application states and check for the different behaviors.
When we search for a product and we see listings, how can we validate the results? How can we be absolutely sure that this data is what we expect? Moreover, how can we make this process repeatable so that on every new deployments we get the same outcome?
How to Make Automated Tests Deterministic
Why do we need controlled data in automated tests?
I give you an example of a bad practice which I frequently see being done in automated tests:
A very poor test is that we search for a product and we assert that we get some listings displayed. We’re not gonna check deep - as long as there are products displayed on the page, then we’re good. INSANE!
Unfortunately, this would leave some questions unanswered:
- How do we know the data is coming from the correct database? Are we connected to a mock server with mocked data?
- How do we know that the data returned is actually what we searched for?
- How do we know the correct number of items are displayed in the search results page?
We need to be able to assert on the results of the tests. The assertions need to make sense and be valid checks.
If we don’t control the data, then we have no way of knowing or checking for the above questions.
To make automated tests deterministic, we need to seed the data ourselves. The automated tests should inject known data and validate the outcome against that data.
If we just rely on the data that is used by other processes and that is subject to change, then our automated tests would not be reliable. We have no way of determining the results.
How to Make Automated Tests Repeatable
Automated tests should run automatically. In a true automated fashion, tests are triggered by a process such as a CI/CD build pipeline which controls the execution and reporting of the tests.
Again, how we handle the data affects the reliability and repeatability of automated tests.
A typical structure for a good automated test is
- 1 - Setup [Create known data]
- 2 - Test [Use the created data]
- 3 - Teardown [Destroy the created data]
Create Test Data
Why do we need to setup the test data every time? Is this not going to slow us down every time we run the tests? Can we not just set it up once and then reuse the same data?
Well, how do we know what happens to the data after a test has finished? Maybe another person’s test has modified or deleted the data?
How can we be sure the next time we want to run the same test, the data is already present for us to use?
Destroy Test Data
Why do we need to destroy the created test data?
Because on the next execution, when we try to create the test data, there’s going to be duplicate data or worse we might get exceptions thrown in the tests.
If we don’t delete the test data and keep recreating random test data, then well, overtime the database is going to have lots of test data and we’d have other problems.
So be sure to create and destroy your test data.
Summary
In order to get the most value from our test automation efforts, we need to design good tests with good structure.
One way that we can make our tests predictable and deterministic is to control the test data. Instead of relying on existing data for testing, the automated tests should seed the data as a pre-step to running scenarios.
By seeding our own test data, we can test for a variety of conditions. Moreover, we can be certain that the assert statements are checking for known data. This will make the tests deterministic.
To run automated test scenarios repeatedly, we need to ensure that our tests create the test data before running the scenarios. This is done in the setup section of an automated test.
The scenarios would then use the data created in the setup step.
Finally when we are done with testing, we should have a way of cleaning the test environment by deleting any created data. This is done in the teardown section of an automated test.