Mutation testing

This page describes how to use mutation testing to improve unit tests.

Setup

We already have a mutation testing tool in the Prosody repo, but you’ll need to make sure you have some of its dependencies installed.

Also, if you haven’t already, you need to have everything you need to compile Prosody. See [

busted

On Debian, apt install lua-busted or on other platforms you can try luarocks install busted if you have luarocks installed.

ltokenp

Next, you’ll need to download, compile and install ltokenp. Don’t worry, it’s easy. Head to this URL and click the ‘5.x’ link to download the source tarball for Lua 5.x.

tar xzf ltokenp-103.tar.gz
cd ltokenp-103
make all

This will create an ltokenp executable in the current directory and test that it works.

To install it, simply copy it to somewhere in your $PATH. You can do this easily (to ‘/usr/local/bin/ltokenp’) with this command:

sudo make install

Generating mutants

We’re going to use ‘util.array’ as an example in this section, but replace it with whichever module you want to test. The tool will automatically identify the correct filename and tests to use from spec/.

./tools/test_mutants.sh.lua util.array

All being well, the tests will run and report the mutation score at the end. Any mutants that were successfully caught by the existing tests will have been deleted. Mutants that were not caught are kept on the disk for inspection.

Dealing with uncaught mutants

List the uncaught mutants:

ls util/array.mutant*.lua

Open one of the mutant files. The source code will be unformatted, but there will also be a comment explaining the change that was made in that mutant.

Now, open up the original file (e.g. util/array.lua) and go to the modified line (which will show you the original non-mutated source code).

The first thing to check is: should this change have caused the tests to fail? Usually the answer is “yes”, but often changes are harmless (this often happens when the tool modifies parameters of a string.sub() or string.byte() call, or causes a performance optimization to be skipped without changing the program behaviour). We call these “equivalent mutants”, and they can be ignored. If you’re not sure, don’t be afraid to ask for a second opinion - it’s not always easy, especially if it’s unfamiliar code.

Once you’re convinced that the mutant introduced a bug that should have been caught by the tests, your challenge is to add extra tests that will catch the mutant. Open up the spec file (e.g. spec/util_array_spec.lua) containing all the tests. Figure out what tests are missing, and add them.

Once you think the new tests are added, re-run the mutation tests to confirm:

./tools/test_mutants.sh.lua util.array

If all is well, the mutation score will have increased. Often, trying to fix one mutant will also fix others too!

Sometimes your extra tests may cause the unmodified source to fail. The tool will warn you if this happens, and refuse to run the mutation tests. You’ll need to check whether your tests are correct, or if maybe you found a bug in the module. If so, congratulations :)

Once only equivalent mutants are remaining for a module, you can move on. Run make test lint, and if everything looks good you can commit your new tests along with any bug fixes, and move on to another module.

If this is your first time working on Prosody, check our contributing guidelines for helpful advice and tips. And thank you!