From CitconWiki
Jump to navigationJump to search

Mutation testing tools: Jester, Jumble & Test Police

Squirrel, Wolf, Eric, Ulrich, Erik, Mieke, Robert, Martijn, Mark, David, Ivan, Kirsten

Jester is a mutation test tool that change the code. If the tests still pass, then it means that tests are missing. Problems: it is hard to integrate it with build scripts, and with multiple tests. Promise of a new release! It is too slow to run on an entire project.

Jumble: similar, but changes bytecode, not source.

  • Problems: difficult to use, not clear how well maintained. Also, Class.getPackage() doesn't return the package name you expect.

some metrics:

  • 5 seconds per compilation
  • 20 mutations per class, each requiring compilation
  • 1200 classes
  • 120000 seconds to run = 33.3 hours

Ajester: similar to Jester, but with bytecode. The one interesting thing is that it is running on the classes that already have a good code coverage.

Martijn: it'd be nice if you could detect tests that always find out the same changes in the code, as tey would be redundant.

Ivan: an "MD5 test" is when a test is almost a rehash of the code, and it makes it harder to change the code.

David: it'd be nice if we could combine Jumble and Emma, and get reports that tell which lines are exercised by which tests.

μJava: a nasty competitor to Jester! does mutation of classes

Iven: there is probably a tool integrated with Maven. Must find it again. Must also look at Hansel (& Gretel)

Tuomas: I'd like to look into infinitest, and see if that could be used to show which lines are exercised by which tests: also, I'm interested in seeing how mutation testing can properly identify which abstract superclasses should be mutated. (I wrote Test Police)

Links to tools