At Mozilla CI Tools, we use Coveralls to track our test coverage progress. In the past month we noticed that sometimes the coverage changes even in PRs that only changed files that are not tracked by coveralls. How come?

Comparing coverage reports for builds in which this problem happens, I noticed that this line would sometimes be marked as covered, sometimes as not covered. The line belongs to a function that builds a graph from a list. An interesting thing about this function is that the order of the list does not affect the final graph, but it does affect the code path taken inside the function. In the test for this function, the input comes from a call to a dict.keys() method. Also, an interesting thing about the .keys() method is that it always produces a list containing the same elements, but the order varies. Mystery solved!

So a single sort() command on the test input could fix those annoying random coverage failures!

tags : Mozilla