Declarative mapping of tests to source

[jayvdb]

The ability to control which source are measured is a great idea, which @coala really needs as we require 100% coverage, but have large test suites and a) developers should only need to run a subset of the tests most of the time, and b) I want to split the test suite across different CI workers, but require they each reach 100% of the appropriate subset.

The current functionality with the hook should be sufficient, however it could be eliminated for many users by having a way to map tests to source with declarative config.

In the case of coala, we use tests/foo/BooTest.py to cover <lib>/foo/Bar.py . That relationship could be described with a regex pattern in the config.

[carljm]

Hi, thanks for the proposal! I’m not quite convinced. I generally use a similar pattern to what you describe, but a) I typically find there are a few annoying edge case exceptions to the pattern, b) such a pattern is easily implemented generically in a few simple lines of hook code, c) I’m not sure how you’d actually describe this mapping using a regex in a way that’s easily comprehensible, and d) the existing hook solution can cover all possible cases. I’m just not convinced the declarative approach is sufficiently better to justify maintaining and documenting two different ways to do it.

If you feel strongly about it, maybe provide an example of how you think the config should look? Will reopen if you want to continue to discuss.

[jayvdb]

The most clear example of when this is beneficial is when each sub-package is a distinct plugin, with separate test modules for each plugin.

See bears vs tests. We have no edge cases in our coala-bears repo (I know, as I have written a shell script to delete the source .py files when splitting the tests across VMs coala/coala-bears#1386 , and I want a better solution for that ugly hack).

Very quick example config

coverage_source_mapping =
  (.*)Test.py : $1.py

As you note, it is easy to implement this using the existing hook approach. I expect adding this declarative mapping would be done by providing an impempentation of that hook implementation which does the pattern matches - should only be a few lines? ;-) For coala it will be the same for many of our repos, so it is worth having it rolled into a plugin so that it is tested separately. Anyways, not critical for us. We have configuration management, so we can force-push the conftest.py to all relevant repos.

[carljm]

I think that regex config isn’t complete, you need to specify the path transform as well (tests to lib dir), but that can be worked in the same format. The other case a hook typically handles is full dir to full dir rather than file to file.

I’m not opposed to adding this if there were a PR for it.