DeprecationWarnings when running tests on Python 3.11

[icemac]

FEATURE REQUEST

What I did:

tox -epy311

What I expect to happen:

Successful test run without any deprecation warnings.

What actually happened:

AuthEncoding-4.3-py3.11.egg/AuthEncoding/AuthEncoding.py:200: DeprecationWarning: 'crypt' is deprecated and slated for removal in Python 3.13
  from crypt import crypt
/Users/mac/vcs/gocept/Zope/src/ZPublisher/HTTPRequest.py:23: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
  from cgi import FieldStorage
AccessControl-5.7-py3.11-macosx-12.0-x86_64.egg/AccessControl/SecurityInfo.py:272: DeprecationWarning: The smtpd module is deprecated and unmaintained and will be removed in Python 3.12.  Please see aiosmtpd (https://aiosmtpd.readthedocs.io/) for the recommended replacement.
  __import__(mname, *imp)
python3.11/smtpd.py:96: DeprecationWarning: The asyncore module is deprecated and will be removed in Python 3.12. The recommended replacement is asyncio
  import asyncore
/python3.11/smtpd.py:97: DeprecationWarning: The asynchat module is deprecated and will be removed in Python 3.12. The recommended replacement is asyncio
  import asynchat

I left out the deprecation warnings about changes in unittest as there is already zopefoundation/meta#143

It seems that we will get into some trouble because Python removes some foundational libraries used by Zope or at least its tests.

What version of Python and Zope/Addons I am using:

• Zope master branch
• Python 3.11.1

[d-maurer]

Michael Howitz wrote at 2023-1-10 06:27 -0800:

... /Users/mac/vcs/gocept/Zope/src/ZPublisher/HTTPRequest.py:23: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13 from cgi import FieldStorage

I think, I have seen already a discussion about a `cgi.FieldStorage` replacement.

AccessControl-5.7-py3.11-macosx-12.0-x86_64.egg/AccessControl/SecurityInfo.py:272: DeprecationWarning: The smtpd module is deprecated and unmaintained and will be removed in Python 3.12. Please see aiosmtpd (https://aiosmtpd.readthedocs.io/) for the recommended replacement. __import__(mname, *imp)

The `smtpd` reference comes from `pages.txt` and `security.pt` in `Products.Five.browser.tests`. It likely can be replaced by any module usually not "allowed".

python3.11/smtpd.py:96: DeprecationWarning: The asyncore module is deprecated and will be removed in Python 3.12. The recommended replacement is asyncio import asyncore /python3.11/smtpd.py:97: DeprecationWarning: The asynchat module is deprecated and will be removed in Python 3.12. The recommended replacement is asyncio import asynchat

Those will go away once the `smtpd` reference is gone.

[dataflake]

Once new releases for AuthEncoding, AccessControl, DocumentTemplate and DateTime are made and the version pins updated the only remaining warning should be about the cgi module.

[d-maurer]

Jens Vagelpohl wrote at 2023-1-11 23:03 -0800:

Once new releases for `AuthEncoding`, `AccessControl`, `DocumentTemplate` and `DateTime` are made and the version pins updated the only remaining warning should be about the `cgi` module.

I plan to work on a `cgi.FieldStorage` replacement -- probably together with work on #648.

[dataflake]

FYI, I have created releases for AuthEncoding, AccessControl, DocumentTemplate and DateTime and updated the version pins on master.

[d-maurer]

Are we ready to depend on https://pypi.org/project/multipart (homepage https://github.com/defnull/multipart) as replacement for cgi.FieldStorage or do we want to implement our own FieldStorage?

[dataflake]

Are we ready to depend on https://pypi.org/project/multipart (homepage https://github.com/defnull/multipart) as replacement for cgi.FieldStorage or do we want to implement our own FieldStorage?

multipart looks OK to me. It doesn't seem to have external dependencies and it's still under (slow) development. The only reason for implementing our own would be if that is very simple to do.

[d-maurer]

Jens Vagelpohl wrote at 2023-1-13 02:28 -0800:

... `multipart` looks OK to me. It doesn't seem to have external dependencies and it's still under (slow) development. The only reason for implementing our own would be if that is very simple to do.

There are 2 main problems: 1. the complexity of MIME messages (which HTTP builds upon) 2. the need to handle really large messages (RAM) efficiently. The first problem could be addressed by using Python's `email.feedparser`. Unfortunately, it builds an in-memory structure and therefore is not RAM efficient for large messages (e.g. for uploads). I looked at the code and it cannot easily be made to avoid large in memory structures. The `multipart` `README` promises efficient handling of large messages. Therefore, I would prefer to give `multipart` a trial.

[d-maurer]

Dieter Maurer wrote at 2023-1-13 12:14 +0100:

Jens Vagelpohl wrote at 2023-1-13 02:28 -0800: > ... >`multipart` looks OK to me. It doesn't seem to have external dependencies and it's still under (slow) development. The only reason for implementing our own would be if that is very simple to do. ... The `multipart` `README` promises efficient handling of large messages. Therefore, I would prefer to give `multipart` a trial.

We cannot use the main entry function of `multipart` (i.e. `parse_form_data`) because it uses (home grown) mappings for its result which do not preserve the form field order. Preserving this order is essential for the `:records` modifier. However, we can use the `multipart` parser to do the `multipart/form-data` parsing - i.e. the most difficult part.

[icemac]

zope.publisher already uses the multipart library in https://github.com/zopefoundation/zope.publisher/blob/master/src/zope/publisher/browser.py It was initially implemented in zopefoundation/zope.publisher#55.

[icemac]

After #1094 is merged, we now only have the following (besides some warnings during installation):

• Fix deprecation warning in tests. ExtensionClass#61
• WebOb-1.8.7-py3.11.egg/webob/compat.py:5: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13 (which is out of our control).