Why I hate test coverage tools

Because it could lie if we abuse it. I have been in one project where people count acceptance tests (written in FIT) so the test coverage always looks pretty high when the unit tests coverage are actually poor. But the figure gave people there false confidence that their code is in good shape.

In another project, a test had only two lines of code, but it actually ran about 5 seconds and touched a lot of code underneath. So by this test alone, it improved the test coverage a lot, looks pretty good eh? Then wait until it failed sometimes. People had absolutely no clue on why it failed and where it caused the failure. and then we could spent one hour to just pinpoint where the problem is. For me, this test is simply just not a test and should be replaced by a bunch of more fine-grained unit tests.

The test coverage number is just a number. It helps when the number is low so that you know you have serious problems in your project. But when the number is high, it doesn't mean anything. We could still do something with it, such as browsing through the nice colored reports to see where we failed to test and if we should fix that. But we should never trust we are in good shape just because the number is high.

To make our test coverage more meaningful, we should remember that TDD is a tool to help us designing our code, to improve the code quality. If we feel awkward in writing a test, then we know there is something wrong in our code and we should refactor that piece, instead of just writing a bad test or just be off.

Published Monday, November 19, 2007 3:42 PM by ywen
Filed under ,

Comments


Tuesday, November 20, 2007 12:19 PM by bret

# re: Why I hate test coverage tools

It is important to remember that code coverage only tells you that the code was exercised -- not that it produced correct results. You can get 100% with a test suite that has no asserts!