Software Testing Fundamentals - Questions and Answers
Software Testing is an activity in software development. It is an investigation performed against a software to provide information about the quality of the software to stakeholders.
What is Software Testing?
Different people have come up with various definitions for Software Testing, but generally, the aim is:
- To ensure that the software meets the agreed requirements and design
- The application works as expected
- The application doesn’t contain serious bugs
- Meets its intended use as per user expectations
Software testing is often used in association with the terms verification and validation.
Validation: Are we doing the right job? Verification: Are we doing the job right?
Verification is the checking or testing of items, including software, for conformance and consistency with an associated specification.
Validation is the process of checking that what has been specified is what the user actually wanted.
Software testing is just one kind of verification, which also uses techniques such as reviews, analysis, inspections and walk-through.
What is Exploratory Testing and when should it be performed?
The definition of Exploratory Testing is “simultaneous test design and execution” against an application. This means that the tester uses her domain knowledge and testing experience to predict where and under what conditions the system might behave unexpectedly. As the tester starts exploring the system, new test design ideas are thought of on the fly and executed against the software under test.
On an exploratory testing session, the tester executes a chain of actions against the system, each action depends on the result of the previous action, hence the outcome of the result of the actions could influence what the tester does next, therefore the test sessions are not identical.
This is in contrast to Scripted Testing where tests are designed beforehand using the requirements or design documents, usually before the system is ready and execute those exact same steps against the system in another time.
Exploratory Testing is usually performed as the product is evolving (agile) or as a final check before the software is released. It is a complementary activity to automated regression testing.
What Test Techniques are there and what is their purpose?
Test Techniques are primarily used for two purposes: a) To help identify defects, b) To reduce the number of test cases.
- Equivalence partitioning is mainly used to reduce the number of test cases by identifying different sets of data that are not the same and only executing one test from each set of data
- Boundary Value Analysis is used to check the behavior of the system at the boundaries of allowed data.
- State Transition Testing is used to validate allowed and disallowed states and transitions from one state to another by various input data
- Pair-wise or All-Pairs Testing is a very powerful test technique and is mainly used to reduce the number of test cases while increasing the coverage of feature combinations.
Why is Testing Necessary?
Testing is necessary in order to identify any defects that are present in software which can cause harm. Without proper testing, we could potentially release a software which could malfunction and cause serious injuries.
Examples can be:
- Software in a life support machine which can cause serious harm to a patient;
- Software in a nuclear plant which monitors nuclear activity can cause harm to the environment
- Banking or financial application which calculates exchange rates can cause financial loss to a business
What is the difference between a Bug, Defect, Error, Failure, Fault, and Mistake?
Error and Mistake are the same things. Bug, Defect, and Fault are the same thing.
In general, a human being can make a mistake (error) which produces a defect (bug, fault) in a software application which may cause a failure.
Defects occur because human beings are prone to make mistakes, also a software application can be very complex so the integration of different components can cause odd behaviors.
How much testing is enough?
There is no definitive answer to this question. Testing is not absolute and has no limits. However, we can use risk metrics (risk-based testing) to identify the likely scenarios that can cause the most harm or the sections of the software that is mostly used so that we focus our time and effort to the sections that are most important.
Testing should provide enough information about the status or health of an application, so the stakeholders can make an informed decision on whether to release the software or spend more time on testing.
What is the Fundamental Test Process
In order to gain the most of the testing activities, a defined process must be followed. But before any testing activity begins, much of the effort should be spent on producing a good test plan. A good test plan goes a long way in ensuring that the testing activities are adhered to what the testing is trying to achieve.
It is perhaps most applicable to a fairly formal testing environment (such as mission critical). Most commercial organisations have less rigorous testing processes. However, any testing effort can use these steps in some form.
The Fundamental Test Process comprises five activities:
- Planning
- Specification
- Execution
- Recording
- Checking for Test Completion
The test process always begins with Test Planning and ends with Checking for Test Completion.
Any and all of the activities may be repeated (or at least revisited) since a number of iterations may be required before the completion criteria defined during the Test Planning activity are met.
Seven Principles of Software Testing
Below are the seven principles of software testing:
1. Testing shows the presence of bugs
Testing an application can only reveal that one or more defects exist in the application, however, testing alone cannot prove that the application is error free. Therefore, it is important to design test cases which find as many defects as possible.
2. Exhaustive testing is impossible
Unless the application under test (AUT) has a very simple logical structure and limited input, it is not possible to test all possible combinations of data and scenarios. For this reason, risk and priorities are used to concentrate on the most important aspects to test.
3. Early testing
The sooner we start the testing activities the better we can utilize the available time. As soon as the initial products, such the requirement or design documents are available, we can start testing. It is common for the testing phase to get squeezed at the end of the development lifecycle, i.e. when development has finished, so by starting testing early, we can prepare testing for each level of the development lifecycle.
Another important point about early testing is that when defects are found earlier in the lifecycle, they are much easier and cheaper to fix. It is much cheaper to change an incorrect requirement than having to change a functionality in a large system that is not working as requested or as designed!
4. Defect clustering
During testing, it can be observed that most of the reported defects are related to small number of modules within a system. i.e. small number of modules contain most of the defects in the system. This is the application of the Pareto Principle to software testing: approximately 80% of the problems are found in 20% of the modules.
5. The pesticide paradox
If you keep running the same set of tests over and over again, chances are no more new defects will be discovered by those test cases. Because as the system evolves, many of the previously reported defects will have been fixed and the old test cases do not apply anymore.
Anytime a fault is fixed or a new functionality added, we need to do regression testing to make sure the new changed software has not broken any other part of the software. However, those regression test cases also need to change to reflect the changes made in the software to be applicable and hopefully fine new defects.
6. Testing is context dependent
Different methodologies, techniques and types of testing is related to the type and nature of the application. For example, a software application in a medical device needs more testing than a games software.
More importantly a medical device software requires risk based testing, be compliant with medical industry regulators and possibly specific test design techniques.
By the same token, a very popular website, needs to go through rigorous performance testing as well as functionality testing to make sure the performance is not affected by the load on the servers.
7. Absence of errors fallacy
Just because testing didn’t find any defects in the software, it doesn’t mean that the software is ready to be shipped. Were the executed tests really designed to catch the most defects? or where they designed to see if the software matched the user’s requirements? There are many other factors to be considered before making a decision to ship the software.
What is White Box Testing
White box testing deals with the internal logic and structure of the code. White box testing is also called as glass, structural, open box or clear box testing. The tests written based on the white box testing strategy incorporate coverage of the code written, branches, paths, statements and internal logic of the code etc.
In order to implement white box testing, the tester has to deal with the code and hence is needed to possess knowledge of coding and logic i.e. internal working of the code. White box test also needs the tester to look into the code and find out which unit/statement/chunk of the code is malfunctioning.
Unit Testing
The developer carries out unit testing in order to check if the particular module or unit of code is working fine. The Unit Testing comes at the very basic level as it is carried out as and when the unit of the code is developed or a particular functionality is built.
Static and dynamic Analysis
Static analysis involves going through the code in order to find out any possible defect in the code. Dynamic analysis involves executing the code and analyzing the output.
Statement Coverage
In this type of testing the code is executed in such a manner that every statement of the application is executed at least once. It helps in assuring that all the statements execute without any side effect.
Branch Coverage
No software application can be written in a continuous mode of coding, at some point we need to branch out the code in order to perform a particular functionality. Branch coverage testing helps in validating of all the branches in the code and making sure that no branching leads to abnormal behavior of the application.
Security Testing
Security Testing is carried out in order to find out how well the system can protect itself from unauthorized access, hacking – cracking, any code damage etc. which deals with the code of application. This type of testing needs sophisticated testing techniques.
Mutation Testing
A kind of testing in which, the application is tested for the code that was modified after fixing a particular bug/defect. It also helps in finding out which code and which strategy of coding can help in developing the functionality effectively.
Advantages of White Box Testing
As the knowledge of internal coding structure is prerequisite, it becomes very easy to find out which type of input/data can help in testing the application effectively. The other advantage of white box testing is that it helps in optimizing the code It helps in removing the extra lines of code, which can bring in hidden defects.
Disadvantages of White Box Testing
As knowledge of code and internal structure is a prerequisite, a skilled tester is needed to carry out this type of testing, which increases the cost. And it is nearly impossible to look into every bit of code to find out hidden errors, which may create problems, resulting in failure of the application.
What is Black Box Testing
In Black Box Testing, the tester tests an application without knowledge of the internal workings of the application being tested.
Because black box testing is not concerned with the underlying code, then the techniques can be derived from the requirement documents or design specifications and hence testing can start as soon as the requirements are written.
Boundary Value Analysis Test Technique
Boundary value analysis, BVA, tests the behavior of a program at the boundaries. When checking a range of values, after selecting the set of data that lie in the valid partitions, next is to check how the program behaves at the boundary values of the valid partitions. Boundary value analysis is most common when checking a range of numbers.
State Transition Technique
State transition test technique is used where some aspect of the system can be described in what is called a “finite state machine”. This simply means that the system can be in a (finite) number of different states, and the transitions from one state to another are determined by the rules of the “machine”.
This is the model on which the system and the tests are based. Any system where you get a different output for the same input, depending on what has happened before, is a finite state system.
Equivalence Partitioning Test Technique
The idea behind the equivalence partitioning test technique is to eliminate the set of input data that make the system behave the same and yield the same result when testing a program.
The process of equivalence partitioning technique involves identifying the set of data as an input condition that give the same result when executing a program and classifying them as a set of equivalent data (because they make the program behave in the same way and generate the same output) and partitioning them from another equivalent set of data.
Advantages of Black Box Testing
- The test is unbiased because the designer and the tester are independent of each other.
- The tester does not need knowledge of any specific programming languages.
- The test is done from the point of view of the user, not the designer.
- Test cases can be designed as soon as the specifications are complete.
Disadvantages of Black Box Testing
- The test can be redundant if the software designer has already run a test case.
- The test cases are difficult to design.
- Testing every possible input stream is unrealistic because it would take a inordinate amount of time; therefore, many program paths will go untested.