Cost effective use of Code Coverage in Software Testing

In previous posts we have explained how a Traceability Matrix can be used to ensure that tests exist for each System Requirement. If those requirements have been well defined, are testable and assuming the test team are suitable qualified this approach will achieve good results. But this alone does not ensure that the tests have reached all parts of the Software Product. This post looks at how code coverage can be used in cost effective way to further improve the quality of the delivered system.

What Is Code Coverage:

Code coverage is measured by a tool that integrates with or is part of the software development environment. It functions to measure the percentage of the application’s source code that has been executed (covered) by testing activities. So if the tool measures an application has 70% code coverage, that means 70% of its code has been executed during testing – and consequently, that 70% should not contain bugs, since it has already been tested fully. The test team will use the results of the code coverage to understand the areas of code that have not been reached and then continue to refine the testing until the required coverage is achieved. In understanding the un-tested paths it also usual to find extra code that isn’t necessary for the application to perform its intended purpose. Extra code left inside an application, obviously means unnecessary risk and therefore should be removed.

Making the process Cost Effective:

So, Code Coverage focuses the test team on understanding each and every line of code in the system and ensuring that it is adequately covered by a test. It is clear that this approach will benefit the quality of the delivered system. The downside of the approach is the additional time required to achieve this. It will involve many iterations of run all tests, examine uncovered paths, refine tests, run all tests again. If the run test cycle is long this will add considerably to timescales and therefore cost. Having said this there are a number of approaches that can be used that significantly improve this. Starting with the code itself most code is written in accordance with documented coding standards, one of the focuses of such a standard should be to write code in a way that is easily testable. Typically code that has been written to be testable also contains less bugs! Testing needs to progress incrementally with the development, a strategy of test everything at end will push out the delivery timescale and take longer. Lastly and most importantly the testing should be automated, an automated approach testing means that the test cycle takes minutes against manual testing which takes hours / days.

Summary:

Code coverage has the benefit of ensuring that all functional areas of a system are considered and tested. It is however a technique that is typically only used in safety related or safety critical systems because of cost. This posts suggests that if the use of code coverage is considered up front the associated costs can be significantly reduced. Those costs then need to be considered against the cost of system downtime in the event of system errors that have not been found during testing.