Test installed version of packages using buildout in tox.ini

[icemac]

Suggestions how to test against an installed package instead against the source for packages using zc.buildout in tox.ini:

https://hynek.me/articles/testing-packaging/ is a good read for the advantages of using src/ layouts.

I too would disagree with "forcing package structure changes just to bend a package to a tool that could add intelligence to work around that instead", but src/ layouts have other advantages, and I think we want them because of those.

I found the article really interesting and investigated a little more. It seems that even among the packages we have that do have a src folder there are probably very few (None?) that correctly handle what I read as the main point: Make sure your tests run over the installed version of the package, not just the source tree.

I looked at DocumentTemplate and experimented for a bit to find a combination that appears to work. Here's what else I needed to change (see https://github.com/zopefoundation/DocumentTemplate/pull/53/files):

• remove develop = . in buildout.cfg
• make sure I tell buildout to not look for a stable egg by specifying versions = versions under [buildout] and adding a [versions] stanza with DocumentTemplate =
• in the [test] buildout config add defaults = ['-sDocumentTemplate'] to make sure the test runner only looks for DocumentTemplate tests. This restriction is needed because the generated test script looks in the tox environment site-packages folder, which may contain other stuff.
• change tox.ini to add a [coverage:paths] stanza and change [coverage:run] so source is now DocumentTemplate instead of src (this is described in https://hynek.me/articles/testing-packaging)
• in tox.ini remove any skip_install = true, which prevents the installation of the egg into the tox environments
• in tox.ini remove buildout:develop={toxinidir} from the buildout invocation
• in tox.ini, add zope.testrunner to the deps in [testenv] - the egg path zc.recipe.testrunner embeds into bin/test wouldn't work anymore

Many of the options I needed to change are common in our packages. So making sure to "do the right thing" is a bit of work. Unless I did this the wrong way. Comments welcome.

Originally posted by @dataflake in zopefoundation/zc.zdaemonrecipe#3 (comment)