MutationTesting

Mutation testing tools: Jester, Jumble & Test Police

Squirrel, Wolf, Eric, Ulrich, Erik, Mieke, Robert, Martijn, Mark, David, Ivan, Kirsten

Jester is a mutation test tool that change the code. If the tests still pass, then it means that tests are missing. Problems: it is hard to integrate it with build scripts, and with multiple tests. Promise of a new release! It is too slow to run on an entire project.

Jumble: similar, but changes bytecode, not source.

• Problems: difficult to use, not clear how well maintained. Also, Class.getPackage() doesn't return the package name you expect.

some metrics:

• 5 seconds per compilation
• 20 mutations per class, each requiring compilation
• 1200 classes
• 120000 seconds to run = 33.3 hours

Ajester: similar to Jester, but with bytecode. The one interesting thing is that it is running on the classes that already have a good code coverage.

Martijn: it'd be nice if you could detect tests that always find out the same changes in the code, as tey would be redundant.

Ivan: an "MD5 test" is when a test is almost a rehash of the code, and it makes it harder to change the code.

David: it'd be nice if we could combine Jumble and Emma, and get reports that tell which lines are exercised by which tests.

μJava: a nasty competitor to Jester! does mutation of classes

Iven: there is probably a tool integrated with Maven. Must find it again. Must also look at Hansel (& Gretel)

Tuomas: I'd like to look into infinitest, and see if that could be used to show which lines are exercised by which tests: http://code.google.com/p/infinitest/ also, I'm interested in seeing how mutation testing can properly identify which abstract superclasses should be mutated. (I wrote Test Police)

Links to tools

• Jumble: [1]
• Jester: [2]
• Test Police: [3]
• MuJava: [4]
• More links on [5]