Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Python 3.10. #988

Merged
merged 10 commits into from
Nov 19, 2021
Merged

Add support for Python 3.10. #988

merged 10 commits into from
Nov 19, 2021

Conversation

icemac
Copy link
Member

@icemac icemac commented Sep 29, 2021

To actually support Python 3.10 we need to fix and release zopefoundation/RestrictedPython#214. The current version of RestrictedPython can only be installed on Python 3.10 because zc.buildout ignores python_requires.

There are some deprecation warnings (e. g. zopefoundation/ZODB#353) which could be fixed, too.
I did not yet create issues for all of them.

Open issues:

The only fixable deprecation warning is zopefoundation/ZODB#353 which is not critical, yet.
There are deprecation warnings in some C code, which are not critical yet:

Copy link
Member

@dataflake dataflake left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the Zope package itself those changes are totally fine, I'm just concerned about RestrictedPython and possible security issues.

@loechel
Copy link
Member

loechel commented Sep 29, 2021

@dataflake RestrictedPython has a very security aware release process:

  • python_requires are limited to supported / checked versions
  • The implemented Node Transformer used in RestricedPython uses a allow-list approach, all new Language features are forbidden by default unless a specific visit_ method is added that handles or allow the feature.
  • a regression test suite that checks all defined language features and shows if there are any changes / Problems with a new version

So there should not be any real security issues upfront. But maybe confusion by releases that only add partial support for a Python version. So it is more about communication than security.

@icemac
Copy link
Member Author

icemac commented Oct 1, 2021

I'd like to keep this PR unmerged until a final release of RestrictedPython supporting Python 3.10 as this is one of the requirements to actually support Python 3.10 for Zope. We should not add a halfhearted Python 3.10 support in a bugfix release of Zope but in a feature release with a bit amount of preparation especially for the dependencies.

@icemac icemac added do not merge enhancement RELEASE BLOCK Issue blocking the release/milestone it is in labels Oct 1, 2021
CHANGES.rst Outdated Show resolved Hide resolved
@dataflake
Copy link
Member

@icemac Do we really need code coverage checking on Windows? I suggest removing that, it seems a waste of GitHub Actions minutes.

@dataflake
Copy link
Member

@icemac Do we really need code coverage checking on Windows? I suggest removing that, it seems a waste of GitHub Actions minutes.

And along with it I don't see any need to test building the docs on Windows or linting there. Both those will just duplicate the tests already run on Linux.

@dataflake
Copy link
Member

The more I think about this PR the more I am bothered by a specific aspect, especially after the discussions about roadmaps and backwards compatibility: Please do not remove Python 3.6 support here.

Removing that support "feels" good because Python 3.6 is EOL soon, that is true. But that is throwing applications that use Zope 5 and Python 3.6 under the bus - for no good technical reason. Keep in mind that many Linux distributions still use Python 3.6.

With the recent roadmap discussions in mind I propose a new policy: When you start a release series with a given set of supported Python versions you're free to add to that set. That's a feature and won't break anyone's setup. But you must not take away from that set of supported versions. That is only allowed for the following major release.

@icemac @mauritsvanrees what do you think?

@d-maurer
Copy link
Contributor

d-maurer commented Oct 31, 2021 via email

@dataflake
Copy link
Member

In my opinion, you could stop testing for it.

Please no half measures. All versions that we claim support for must be tested.

@d-maurer
Copy link
Contributor

d-maurer commented Nov 1, 2021 via email

@mauritsvanrees
Copy link
Member

Zope 5 is used by Plone 6, which we only test on Python 3.7-3.9. So from a Plone standpoint it is no problem to drop support for 3.6.
But for other users, it would be nice to only drop it if it is really needed. And I agree, for clarity it would then be better to make this incompatibility clear by doing a new major Zope release.

Let me tell how we handled this in Plone 4:

  • In Plone 4.0 (long time ago) we started with support for only Python 2.6.
  • In Plone 4.2 we supported 2.6 and 2.7.
  • Plone 4.3 started the same
  • Around Plone 4.3.17 we stopped officially supporting Python 2.6. Biggest reason was that it was getting harder and harder to run all tests on Python 2.6.
  • We tried to not actively break 2.6 support, as would for example happen when you use {} instead of {0} in string substitutions. If such breakage accidentally would have landed in a Plone 4.3 version, I would have considered this a bug, and released a new version with a fix.

Danger for Zope 5 is that more and more Python versions will need to be supported because we have no other reason yet to create a breaking Zope release, until one day we say it is enough.
Is that reason enough for creating a Zope 6? You can say yes. That is what Zope 5 did: drop support for Py 2.7, 3.5 and ZServer, so we have a new version.

I just checked Django:

  • Django 3.0.0 and 3.1.0 have classifiers for 3.6-3.8
  • Django 3.2.0 has 3.6-3.9
  • Django 3.2.9 has 3.6-3.10
  • Django 4.0b1 has 3.8-3.10

O, I could have checked the docs for that...

For Plone 6, I can see us removing support for older Python versions in a minor release. We want to do a minor release once a year (which is faster than in the past) so this means we support 3 or 4 Python releases at the same time.

I wonder if this idea might help: in tox.ini add a checker for both the oldest and newest Python version that does basic syntax checking and import checking, without yet running tests. Preferably the newest Python version would be picked automatically when a new one is released, so there is no need to explicitly run the meta package update on all repos.
The idea is then that if both checkers pass, there is at least a good chance that the package works with those versions, even if tests may need to be fixed for the newest version, which can then be tricky to keep running for the oldest version.

I am probably not very clear, you can go multiple ways with this answer...
Let me pick a side:

  • Keep support for as long as we can, otherwise we quickly have three major Zope versions that we would need to support, at least when we want to keep current Plone versions supported.
  • If a newer Python version requires breaking changes that cannot be solved with a try/except ImportError or similar easy fix: yes, create a new major release.

@dataflake
Copy link
Member

This all is getting a bit out of hand and is getting way too complicated. All I am asking for is reverting those changes that drop Python 3.6, including the dropped tests, and that in the future dropping Python versions should only be done in a new major release. And that we stay consistent and test all Python versions we claim to support. We don't need any version recognition magic if we just continue running the tests.

@icemac
Copy link
Member Author

icemac commented Nov 2, 2021

I agree with @dataflake: If we support a version we should test it. Currently we use check-python-versions which expects that the versions allowed by python_requires align with the trove classifiers. So we either have to drop using check-python-versions or keep python_requires and classifiers in sync.

I have a problem with keeping Python 3.6 support until the release of Zope 6: There is no Zope 6 on the horizon and the Python core developers will make relatively sure that supporting newer Python versions will not require to drop support for older versions. (Currently we are even able to run Python 2.7 and 3.10 in the same code base (not for Zope but for all of its dependencies).)
I think we need to have a timeframe when to drop support for older Python versions: I could imagine to say 2 years after a Python version is no longer officially supported by the Python developers we start dropping support for it in the Zope ecosystem. This gives users enough time to migrate, especially because updating the Python version is usually no pain.
Using this approach we would not have to make a major release for dropping Python 3.6 support by the end of 2023 because it is really outdated by that date.

What do you think about this idea?

@dataflake
Copy link
Member

@icemac That is a good compromise. So for the moment we'll return everything to the way it was before (trove classifiers, python_requires, test configurations) and I can add a note to the roadmap with a specific date.

IMHO it's perfectly fine to schedule a major release without big features in order to clean up the supported Python versions. We just need to signal very clearly how long the older Zope are supported so we have light at the end of the (support) tunnel for ourselves.

@dataflake
Copy link
Member

I have changed this branch to omit any changes related to Python 3.6 and updated the wording at https://www.zope.org/developer/roadmap.html to reflect Michael's suggestion.

@icemac icemac self-assigned this Nov 19, 2021
@icemac icemac marked this pull request as draft November 19, 2021 10:48
@icemac icemac marked this pull request as ready for review November 19, 2021 13:40
@icemac
Copy link
Member Author

icemac commented Nov 19, 2021

@dataflake I think this PR is ready now, do you agree in merging it?

@icemac icemac added this to the 5.4 milestone Nov 19, 2021
@dataflake
Copy link
Member

I agree, thanks for doing the work!

@icemac icemac merged commit adacc41 into master Nov 19, 2021
@icemac icemac deleted the config-with-zope-product branch November 19, 2021 14:51
@icemac
Copy link
Member Author

icemac commented Nov 19, 2021

Thank you for reviewing this PR. 😃

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement RELEASE BLOCK Issue blocking the release/milestone it is in
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants