Infrastructure to change many repositories at once

[icemac]

As the Python release manager plans to shorten the Python release cycle to one year (source) we should get some tools to add remove support for python versions from/to packages.

ghcloneall could be a starting point to get all (>300) repos of the zopefoundation.

[mgedmin]

I have a script to verify that the lists of supported versions are consistent between setup.py classifiers, python_requires (if specified), tox.ini, appveyor.yml, and manylinux-install.sh.

[icemac]

@mgedmin Cool, this tool is nice to make the supported versions even. Did you already run it against the zopefoundation repositories? What was the outcome?

[mgedmin]

Here's a full run: https://gist.github.com/mgedmin/52f6a14da53ce66f58f6a1c08803eb0b
and here are the mismatches: https://gist.github.com/mgedmin/52f6a14da53ce66f58f6a1c08803eb0b#file-mismatches-txt

mismatch in AccessControl AuthEncoding cipher.session cmf.pt cmf.zmiicons five.grok five.pt grokcore.content grokcore.errorview grokcore.formlib grokcore.permission grokcore.registries grokcore.resource grokcore.site grokcore.xmlrpc grokproject groktoolkit grokui.admin grokui.base importchecker js.codemirror keas.build keas.unit lovely.rating lovely.recipe manuel megrok.menu megrok.strictrequire nt_svcutils Persistence plone.postpublicationhook plone.validatehook plone.z3cform Products.CMFDefault Products.DCWorkflow Products.MIMETools Products.ZopeVersionControl Record RestrictedPython z3c.authenticator z3c.currency z3c.encryptedpersistent z3c.formdemo z3c.json z3c.jsonrpc z3c.jsonrpcproxy z3c.jsontree z3c.layer.ready2go z3c.language.negotiator z3c.language.session z3c.language.switch z3c.macro z3c.menu.ready2go z3c.recipe.i18n z3c.recipe.paster z3c.schema2xml z3c.soap z3c.sqlalchemy z3c.table z3c.tabular z3c.taskqueue z3c.testsetup z3c.viewtemplate z3c.xmlhttp z3c.zrtresource zc.configuration zc.i18n zc.metarecipe zc.monitor zc.recipe.cmmi zc.recipe.deployment zc.recipe.testrunner zc.table zc.zkzeo zc.zodbwsgi zc.zservertracelog ZEO zmi.icons Zope zope.app.applicationcontrol zope.app.authentication zope.app.broken zope.app.error zope.app.exception zope.app.http zope.app.interface zope.app.principalannotation zope.app.publication zope.app.zcmlfiles zope.app.zopeappgenerations zope.cachedescriptors zope.errorview zope.fixers zope.index zope.locking zope.mimetype zope.password zope.paste zope.wfmc zope.wineggbuilder

[mgedmin]

(Fix for zope.mimetype: zopefoundation/zope.mimetype#16)

[icemac]

@mgedmin Thank you! This is an interesting list. Maybe we have to set some repositories into archive mode because no-one cares about them any more.

[mgedmin]

I should mention that I've published my script to PyPI as check-python-versions.

[icemac]

Maybe https://github.com/asottile/all-repos could also help.

[dataflake]

That looks very promising.

[mgedmin]

https://github.com/jelmer/silver-platter could also be interesting

[mgedmin]

You can now automate updating supported Python versions with

check-python-versions --drop 3.4 .

It'll show you diffs and ask for confirmation before writing any changes to disk. You'll have to update the changelog and do a git commit/pull request dance yourself.

There are more possibilities, e.g.

check-python-versions --drop 2.6,3.0-3.4 --add 3.5-3.7

would allow you to normalize very dusty packages that still support 2.6 and 3.3; --diff can let you look at the diff without having to dismiss all those confirmation prompts, etc. There's more documentation on the PyPI page.

[icemac]

As Python adopts a 12 month release cycle (see https://lwn.net/Articles/803679/), I see more pressure to automate adding new versions and removing old ones.

[mgedmin]

Travis CI has a new feature that might allow us to reduce duplication in our .travis.yml files: https://blog.travis-ci.com/2019-11-11-build-config-imports

[jamadden]

Travis CI has a new feature that might allow us to reduce duplication in our .travis.yml files:

That is very cool. It could work right away for projects without C extensions and manylinux builds. For manylinux builds, we have shell scripts in each repo that also need updating. Perhaps we could embed those in the YAML and echo them out to disk.

[icemac]

See zopefoundation/zope.app.appsetup#4 where I tried the TravisCI import feature.

[icemac]

It looks fine. Seems we need different configs for some packages e. g. the ones having tested documentation. So looking into each package is necessary to decide which config should be used.
I'll start looking into the ZF packages now. Starting with the oldest ones meaning that some of the will get archived instead of tested.

[jugmac00]

This sounds really great!

While this eliminates the duplication between repositories, we still have the duplication between tox.ini and .travis.yml.

What I recently started doing for my own repositories... I just install and call tox in my .travis.yml configurations. This has at least two advantages. Both test runs do the same, and by locally executing tox you can be pretty sure Travis will work (TM).

Any opinions on this?

[dataflake]

That has been the normal procedure for most packages that see regular updates...

[mgedmin]

I was originally very impressed with tox-travis, but received pushback on ZF repos due to custom jobs (docs, pure-python etc). I'm more familiar with Travis CI today and I don't think that's a problem (worst case, you can always specify a custom script in a matrix job, if a custom TOXENV is not sufficient).

(Also, on my personal projects I prefer not to have coverage tracking in tox, for local development iteration speed, but I do want it on Travis. A quick grep shows this is not a concern for the majority of ZF repos.)

[jugmac00]

That has been the normal procedure for most packages that see regular updates...

Maybe, but this blueprint does not call tox.

meta/travis-ci-config/minimal.yml

Lines 1 to 20 in f4d90f4

	language: python
	python:
	- 2.7
	- 3.5
	- 3.6
	- 3.7
	- 3.8
	- pypy
	- pypy3
	before_install:
	- pip install -U pip setuptools
	- pip install -U coverage coveralls
	install:
	- pip install -U -e .[test]
	script:
	- coverage run -m zope.testrunner --test-path=src

I prefer not to have coverage tracking in tox, for local development iteration speed

When I develop, I run tox with one env, and only all incl. cov at the end. Additionally, I use pre-commit hooks. Well, it is all a matter of personal taste.

[dataflake]

I don't even know that "blueprint", I look at actual maintained projects.

[jugmac00]

Maybe I get this wrong, but for me it looks like @icemac is about applying this blueprint for all ZopeFoundation projects? Starting with some old ones...

[dataflake]

I didn't watch closely enough, way too much on my plate right now.

@icemac Will there be a sharable configuration using tox that "normal" Zope product packages re-use?

[icemac]

@dataflake Yes there will be a first shareable version (hopefully by today).

[icemac]

Some suggestions of @glyph in https://glyph.twistedmatrix.com/2020/02/modules-for-maintenance.html we could discuss to use in the ZopeFoundation projects, too.

[jugmac00]

While @glyph's blog post is about tools, I also recommend @hynek's talk "Maintaining a Python Project When It’s Not Your Job" which is also about best practices to maintain floss projects.

Video:
https://www.youtube.com/watch?v=9G2s1TN9QQY

Article:
https://hynek.me/talks/python-foss/

When I have a look at @glpyh's list, we already use a lot of those tools.

As I both use bandit and pre-commit for my personal / work projects, and this combination does not seem to work well (when run via pre-commit, bandit does not pick up its configuration), I wanted to have a look how @glyph configures this in e.g. https://github.com/twisted/twisted - turns out neither pre-commit nor bandit is used (no offense).

One major downside of pre-commit is that we won't see any more please flake8 commits :-) as you cannot commit unless the configured pre-commit hooks are succesfull (unless you override it). I started using it for all of my projects.

Update
There was already a discussion about introducing black: #6

There is another Python security checker, which claims to have all the checks bandit has plus some more: https://github.com/tonybaloney/pycharm-security

As the name suggests, it is a plugin for pycharm, but also can be run headless in ci / docker.

[dataflake]

There were some great links in those articles. I especially liked https://writing.jan.io/2017/03/06/sustainable-open-source-the-maintainers-perspective-or-how-i-learned-to-stop-caring-and-love-open-source.html.

[glyph]

@jugmac00 FWIW, that list of tools was the full list of everything that I've put together one all the projects I've done build automation on, which includes dozens of projects, mostly open source, but definitely some commercial ones as well. Part of the point of the article was that there's no single repo where I've managed to set up absolutely everything correctly.

If anything, Twisted lags a bit behind everything else, since it's dragging a longer tail of legacy & workflow around behind it.

[jugmac00]

Today, I used all-repos twice on the Zope repositories...
The first time to update all .travis.yml files for Python 3.9 final, the second time to update the URLs to the Travis badges in the readme files.

As suggested by @mgedmin in #37, here is again the link to my write up
https://github.com/jugmac00/til/blob/master/zope/how-to-update-all-zopefoundation-repositories-at-once.md

P.S.: As an addition, and as @mgedmin asked in one comment for the many PRs...
Yes, it would be possible to directly push to master, see https://github.com/asottile/all-repos#all_repospushmerge_to_master

Before there is the need to apply some mass updates again, it would make sense to discuss whether it is ok to push trivial changes directly to master, and if so, who got the rights to do this (maybe the core maintainers?).

[dataflake]

Branch protection rules exist on a number of repositories. Most of those only protect master, very few have rules for other branches. The rules are used to e.g. enforce at least one passing PR review or enforce that certain CI tests must pass before a PR can be merged. Only admin-level users can override that and merge in case such a rule prevents merging. I had to do that many times yesterday because I did not want to click around even more to create a positive review, or to get around failing tests in completely unrelated areas.

We have a ton of repositories and I personally am a bit scared of letting some script make all those changes and push it all to master without further review - unless the change is very specific and localized to very few (one?) specific file(s).

I'd be against removing those branch protection rules just to make such sweeping changes easier. Doing those changes across many repositories is or should be a rare exception. I don't mind the extra review/merge work.

[jugmac00]

interim result of mostly @icemac's relentless efforts

repos with .meta.toml == updated with the config tool

❯ all-repos-find-files -C all-repos-zope.json .meta.toml | wc -l
110

repos without .meta.toml

❯ all-repos-list-repos --output-paths -C all-repos-zope.json | xargs --replace bash -c 'ls {}/.meta.toml >& /dev/null || echo {}'| wc -l
178

[icemac]

So not even half of the repositories are migrated to GHA …
There is an upcoming sprint in April where we want to tackle more of those problems, see https://blog.gocept.com/2021/03/08/earl-zope-invites-to-sprint/.

[jugmac00]

repos without .meta.toml

❯ all-repos-list-repos -C zope.json --output-paths | xargs --replace bash -c 'ls {}/.meta.toml >& /dev/null || echo {}' | wc -l
110

repos which still need to be converted

❯ all-repos-list-repos -C zope.json --output-paths | xargs --replace bash -c 'ls {}/.meta.toml >& /dev/null || echo {}'
output_zope/zopefoundation/ZODB
output_zope/zopefoundation/BTrees
output_zope/zopefoundation/persistent
output_zope/zopefoundation/zope.event
output_zope/zopefoundation/zope.i18nmessageid
output_zope/zopefoundation/zope.deprecation
output_zope/zopefoundation/zope.component
output_zope/zopefoundation/zope.copy
output_zope/zopefoundation/zope.hookable
output_zope/zopefoundation/zope.location
output_zope/zopefoundation/zope.proxy
output_zope/zopefoundation/zope.security
output_zope/zopefoundation/ZEO
output_zope/zopefoundation/zope.dottedname
output_zope/zopefoundation/zope.lifecycleevent
output_zope/zopefoundation/zope.filerepresentation
output_zope/zopefoundation/zope.annotation
output_zope/zopefoundation/zope.i18n
output_zope/zopefoundation/zope.container
output_zope/zopefoundation/zope.publisher
output_zope/zopefoundation/zope.traversing
output_zope/zopefoundation/zodbpickle
output_zope/zopefoundation/ZConfig
output_zope/zopefoundation/zc.lockfile
output_zope/zopefoundation/zope.cachedescriptors
output_zope/zopefoundation/zope.contenttype
output_zope/zopefoundation/zope.size
output_zope/zopefoundation/zope.minmax
output_zope/zopefoundation/z3c.coverage
output_zope/zopefoundation/zope.browserresource
output_zope/zopefoundation/zope.session
output_zope/zopefoundation/zope.error
output_zope/zopefoundation/zope.password
output_zope/zopefoundation/zope.pluggableauth
output_zope/zopefoundation/zope.securitypolicy
output_zope/zopefoundation/zope.processlifetime
output_zope/zopefoundation/zope.principalregistry
output_zope/zopefoundation/zope.applicationcontrol
output_zope/zopefoundation/zope.login
output_zope/zopefoundation/zope.browserpage
output_zope/zopefoundation/zope.contentprovider
output_zope/zopefoundation/z3c.form
output_zope/zopefoundation/z3c.template
output_zope/zopefoundation/zope.keyreference
output_zope/zopefoundation/zope.intid
output_zope/zopefoundation/zope.viewlet
output_zope/zopefoundation/zope.structuredtext
output_zope/zopefoundation/zope.principalannotation
output_zope/zopefoundation/zope.index
output_zope/zopefoundation/zope.copypastemove
output_zope/zopefoundation/zope.componentvocabulary
output_zope/zopefoundation/z3c.macro
output_zope/zopefoundation/z3c.pt
output_zope/zopefoundation/z3c.ptcompat
output_zope/zopefoundation/roman
output_zope/zopefoundation/zope.testbrowser
output_zope/zopefoundation/z3c.table
output_zope/zopefoundation/AccessControl
output_zope/zopefoundation/Acquisition
output_zope/zopefoundation/zope.ptresource
output_zope/zopefoundation/Persistence
output_zope/zopefoundation/zope.generations
output_zope/zopefoundation/zope.app.apidoc
output_zope/zopefoundation/Record
output_zope/zopefoundation/zExceptions
output_zope/zopefoundation/ZopeUndo
output_zope/zopefoundation/zope.app.server
output_zope/zopefoundation/zope.deferredimport
output_zope/zopefoundation/zope.mimetype
output_zope/zopefoundation/zope.ramcache
output_zope/zopefoundation/RestrictedPython
output_zope/zopefoundation/z3c.password
output_zope/zopefoundation/keas.profile
output_zope/zopefoundation/grok
output_zope/zopefoundation/grokcore.site
output_zope/zopefoundation/martian
output_zope/zopefoundation/z3c.autoinclude
output_zope/zopefoundation/zc.catalog
output_zope/zopefoundation/zope.vocabularyregistry
output_zope/zopefoundation/five.formlib
output_zope/zopefoundation/z3c.checkversions
output_zope/zopefoundation/z3c.jbot
output_zope/zopefoundation/z3c.rml
output_zope/zopefoundation/five.taskqueue
output_zope/zopefoundation/bobo
output_zope/zopefoundation/zope.outputchecker
output_zope/zopefoundation/zope.app.i18n
output_zope/zopefoundation/zope.app.security
output_zope/zopefoundation/z3c.pdftemplate
output_zope/zopefoundation/groktoolkit
output_zope/zopefoundation/hurry.query
output_zope/zopefoundation/zc.relation
output_zope/zopefoundation/zc.zdaemonrecipe
output_zope/zopefoundation/five.grok
output_zope/zopefoundation/z3c.currency
output_zope/zopefoundation/zc.zlibstorage
output_zope/zopefoundation/z3c.sqlalchemy
output_zope/zopefoundation/zc.zservertracelog
output_zope/zopefoundation/z3c.relationfield
output_zope/zopefoundation/zopetoolkit
output_zope/zopefoundation/zc.zrs
output_zope/zopefoundation/zc.zope3recipes
output_zope/zopefoundation/Products.CMFDefault
output_zope/zopefoundation/ZServer
output_zope/zopefoundation/www.zope.org
output_zope/zopefoundation/zpt-docs
output_zope/zopefoundation/zopefoundation.github.io
output_zope/zopefoundation/meta
output_zope/zopefoundation/website-zope.de
output_zope/zopefoundation/.github

(a couple of above listed repos are no source code repositories)

converted repos

❯ all-repos-find-files -C zope.json .meta.toml | wc -l
177