MutationTesting
Mutation testing tools: Jester, Jumble & Test Police
Squirrel, Wolf, Eric, Ulrich, Erik, Mieke, Robert, Martijn, Mark, David, Ivan, Kirsten
Jester is a mutation test tool that change the code. If the tests still pass, then it means that tests are missing. Problems: it is hard to integrate it with build scripts, and with multiple tests. Promise of a new release! It is too slow to run on an entire project.
Jumble: similar, but changes bytecode, not source.
- Problems: difficult to use, not clear how well maintained. Also, Class.getPackage() doesn't return the package name you expect.
some metrics:
- 5 seconds per compilation
- 20 mutations per class, each requiring compilation
- 1200 classes
- 120000 seconds to run = 33.3 hours
Ajester: similar to Jester, but with bytecode. The one interesting thing is that it is running on the classes that already have a good code coverage.
Martijn: it'd be nice if you could detect tests that always find out the same changes in the code, as tey would be redundant.
Ivan: an "MD5 test" is when a test is almost a rehash of the code, and it makes it harder to change the code.
David: it'd be nice if we could combine Jumble and Emma, and get reports that tell which lines are exercised by which tests.
μJava: a nasty competitor to Jester! does mutation of classes
Iven: there is probably a tool integrated with Maven. Must find it again. Must also look at Hansel (& Gretel)
Tuomas: I'd like to look into infinitest, and see if that could be used to show which lines are exercised by which tests: http://code.google.com/p/infinitest/ also, I'm interested in seeing how mutation testing can properly identify which abstract superclasses should be mutated. (I wrote Test Police)
