comments (not for humans)
O3tker twittered about a framework Microsoft has created called Pex for automatically generating unit tests with high test coverage. There is a video of it here: http://channel9.msdn.com/posts/Peli/Getting-started-with-Pex-in-Visual-Studio-2008/.

The basic features of Pex is that you run it from within Visual Studio 2010, and it will analyze and run through your code and try to detect interesting values based on the branches etc. in the code it is testing. I will display a list of interesting values, and you can choose to save these values as unit tests.

I really like the exploratory part of it. Showing a list of interesting edge cases etc. can really be helpful. My gut reaction though was that I would be very skeptical about saving the tests as unit tests directly. And what is displayed in the video, basically proved the point.

First of all, the generated file says "Do not modify". If I don't modify that file, I'm stuck with tests with names like "Capitalize09". This is a horrible test name. I should be able to tell from the name of a breaking tests what is wrong with my code. Capitalize09 doesn't tell me anything. Next, the generated test suite is bound to be fragile, because the tests know too much about the implementation. Tests tend to be fragile when they are not based on requirements/specification. So this really doesn't fit the BDD-sphere.

Adding tests just to increase coverage, is also not the way to go. Coverage is not a measure on the amount of code that has been tested. It only indicates which code has not been covered by tests. And if you are doing test-first, the code coverage should come from your specifications (tests).

Does that mean I would never use Pex? No, definitely not. I think the exploratory part is really interesting and can be of great value in discovering flaws in the logic. I would however use Pex only to discover the interesting edge cases, and rather use what it discovers as a basis for writing good tests with meaningful names

Comments closed for this post