PEP 771: Amendments following pre-publish discussion #4238

astrofrog · 2025-01-29T11:44:07Z

There has been an extensive discussion going on in this DPO thread while the original PR for PEP 771 was being reviewed. Rather than continuously update that PR, @pradyunsg suggested that that PEP should be published as-is initially and that changes based on the pre-publish discussion should be made in a separate PR. This is the separate PR 🎉

The main changes are:

Real-life examples of packages have been added in several places by popular demand
Most of the 'How to teach this' section has been moved to a new 'Examples of usage' section, and 'How to teach this' has been filled with new content
The backward-compatibility section has been expanded to address concerns raised during the discussion
An open issue has been added as it is an unresolved point that will require more discussion
The default-optional-dependencies key has been renamed to default-optional-dependency-keys

Once this PR is merged, I'll then open the main discussion thread and open a trivial PR to add the link to the PR.

cc @pradyunsg @DEKHTIARJonathan

📚 Documentation preview 📚: https://pep-previews--4238.org.readthedocs.build/

…amples in each sub-section

AA-Turner

Brief review, I have only read up to the changes to Specification in depth.

Edit: Please can you add the linked thread to Post-History?

Post-History:
  `15-Jan-2025 <https://discuss.python.org/t/77892/>`__,

peps/pep-0771.rst

AA-Turner · 2025-01-29T14:24:41Z

peps/pep-0771.rst

+* `astropy <https://www.astropy.org/>`_, a core package for astronomy, which
+  recommends that users install ``astropy[recommended]``. Some of the
+  dependencies in ``recommended``, such as `scipy <http://www.scipy.org>`_, are
+  used in a number of places in astropy for common
+  functionality and scipy should ideally be a required dependency, but there has
+  been reluctance to so as it is not a lightweight dependency and advanced users
+  who want minimal installs want the ability to opt out of it (however, as
+  described above, this places a burden on the majority of regular uses to
+  satisfy a use case for advanced users).
+* `fastapi <https://fastapi.tiangolo.com/>`_, a web framework, which recommends
+  that users install ``fastapi[standard]`` in order to include commonly needed
+  dependencies.
+* `tensorflow <https://www.tensorflow.org>`_, a machine learning platform, which
+  recommends that users install ``tensorflow[and-cuda]`` since CUDA support is
+  commonly needed by users.


This section feels a bit wordy, and repeats the prose description from this section. Perhaps you could list the packages, and say that these fit the description in the paragraph above?

E.g.

There are several packages that demonstrate this pattern by encouraging users to include extra dependencies by default, such as: * (link to astropy): ``astropy[...]``

AA-Turner · 2025-01-29T14:25:31Z

peps/pep-0771.rst

+Real-world examples of packages that allow users to select different backends or
+frontends with extras include:


Likewise, I wonder if there's a better (more concise) way of integrating the description of the pattern with the examples.

peps/pep-0771.rst

AA-Turner · 2025-01-29T14:30:48Z

peps/pep-0771.rst

+Examples of usage
 =================


Please stick to the suggested sections in PEP 12. Perhaps you could add "examples of usage" as a sub-section, but "how to teach this" is broader than simply stating examples.

Suggested change

Examples of usage

=================

How to teach this

=================

I moved the examples into the Specification section

Co-authored-by: Adam Turner <[email protected]>

…he specification

astrofrog · 2025-01-30T10:43:23Z

@AA-Turner - thanks for the review so far! I've added the Post-History as requested (and have implemented the other comments). I also moved the examples section into a sub-section of the specification (as opposed to how to teach this) since it felt more natural as the last subsection of the specification, and there is already an extensive 'how to teach this' - however if you feel strongly this should be a subsection of 'how to teach this' I can move it, so just let me know.

willingc

Thanks for taking this on @astrofrog. I have several comments and questions after reading the PEP but not all of the DPO discussion. I agree with the Motivations, but I'm wrestling a bit if we're communicating the approach clearly enough. I'm not advocating changing the approach but making it more explicit for readers.

willingc · 2025-01-30T22:14:46Z

peps/pep-0771.rst

@@ -70,18 +82,31 @@ or backend package. There is currently no mechanism in Python packaging
 infrastructure to disallow conflicting or incompatible extras to be installed,
 and this PEP does not change that.

+Examples of packages that require at least one backend or frontend to work and


Thanks for bring up this @astrofrog. This has been a usability pain point for a long while.

willingc · 2025-01-30T22:18:11Z

peps/pep-0771.rst


-the ``extra1`` dependency would be included. If the user instead uses::
+the ``extra1`` dependency would be included. If the user instead uses:


Suggested change

the ``extra1`` dependency would be included. If the user instead uses:

the default ``extra1`` dependency would be included. If the user instead uses:

willingc · 2025-01-30T22:20:07Z

peps/pep-0771.rst

-    pip install package[extra2]
+.. code-block:: console
+
+    $ pip install package[extra2]


I recommend updating line 196 to be explicit that the non-default extra2 would be installed, but the default extra1 would not be installed.

willingc · 2025-01-30T22:22:47Z

peps/pep-0771.rst


-    pip install package package[extra2]
+    $ pip install package package[extra2]


It may make sense to include a sample dependency tree here since I think that is far more likely than the above pip command.

The first time I read this my eyes skipped over the first package in the command.

willingc · 2025-01-30T22:25:08Z

peps/pep-0771.rst


 should install the default extras.

 Note that ``package[]`` would continue to be equivalent to ``package`` and would
 not be provided as a way to install without default extras (see the `Rejected
 Ideas`_ section for the rationale).

+We also note that some tools such as `pip`_ currently ignore unrecognized
+extras, and emit a warning to the user to indicate that the extra has not been


...unrecognized non-default extras ???

willingc · 2025-01-30T22:58:18Z

peps/pep-0771.rst

+  extras further down the road, as it will allow ``package[minimal]`` to work
+  for versions prior to when defaults were adopted.
+
+Avoiding adding too many default dependencies


Suggested change

Avoiding adding too many default dependencies

Avoiding the addition of many default dependencies

willingc · 2025-01-30T23:00:25Z

peps/pep-0771.rst

+incompatible. In this case, we recommend against making using the default extra
+feature for any extra that contains a dependency that could be incompatible with
+another.


Suggested change

incompatible. In this case, we recommend against making using the default extra

feature for any extra that contains a dependency that could be incompatible with

another.

incompatible. In this case, we recommend against using the default extra

feature for any extra that contains a dependency that could be incompatible with

another.

willingc · 2025-01-30T23:01:42Z

peps/pep-0771.rst

+
+.. code-block:: console
+
+    $ pip install package


Perhaps add a comment after the command to specify what is being installed (A as a default)

willingc · 2025-01-30T23:02:54Z

peps/pep-0771.rst

+Package users
+-------------
+
+Package users should be provided with clear installation instructions that show


As a napari developer that deals with qt versions and dependencies, clarity is key as a user.

willingc · 2025-01-30T23:07:33Z

peps/pep-0771.rst

+  will need to carefully consider how to treat default extras, and this may
+  imply a non-negligible amount of work and discussion.
+
+It is impossible for a PEP such as this to exhaustively consider each of the


Hmm... @astrofrog, do you expect the impact to be felt most by package maintainers, packaging tool providers, packaging repositories, or package users? It's not clear to me.

astrofrog added 5 commits January 28, 2025 12:28

Added specific examples of packages in the motivation section

6055136

Rename default-optional-dependencies to default-optional-dependency-keys

5898fa9

Rename 'How to teach this' to 'Examples of Usage' and add concrete ex…

14f8560

…amples in each sub-section

Expand backward compatibility and how to teach this

fa12de7

Add note about unrecognized extras

c93dac5

astrofrog requested a review from pradyunsg as a code owner January 29, 2025 11:44

Tweak link

da37547

astrofrog changed the title ~~PEP 771 - edits following pre-publish discussion~~ PEP 771: edits/additions following pre-publish discussion Jan 29, 2025

astrofrog changed the title ~~PEP 771: edits/additions following pre-publish discussion~~ PEP 771: Edits/additions following pre-publish discussion Jan 29, 2025

AA-Turner reviewed Jan 29, 2025

View reviewed changes

AA-Turner changed the title ~~PEP 771: Edits/additions following pre-publish discussion~~ PEP 771: Amendments following pre-publish discussion Jan 29, 2025

astrofrog and others added 4 commits January 30, 2025 10:21

Apply suggestions from code review

eed0a98

Co-authored-by: Adam Turner <[email protected]>

Make examples into a sub-section of the specification

600da1c

Make examples more concise, and move examples into a sub-section of t…

f7cc0c8

…he specification

Fixed link

f0bbf2f

willingc reviewed Jan 30, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PEP 771: Amendments following pre-publish discussion #4238

PEP 771: Amendments following pre-publish discussion #4238

astrofrog commented Jan 29, 2025 •

edited by github-actions bot

Loading

AA-Turner left a comment •

edited

Loading

AA-Turner Jan 29, 2025

astrofrog Jan 30, 2025

AA-Turner Jan 29, 2025

astrofrog Jan 30, 2025

AA-Turner Jan 29, 2025

astrofrog Jan 30, 2025

astrofrog commented Jan 30, 2025

willingc left a comment •

edited

Loading

willingc Jan 30, 2025

willingc Jan 30, 2025

willingc Jan 30, 2025

willingc Jan 30, 2025

willingc Jan 30, 2025

willingc Jan 30, 2025

willingc Jan 30, 2025

willingc Jan 30, 2025

willingc Jan 30, 2025

willingc Jan 30, 2025

		Real-world examples of packages that allow users to select different backends or
		frontends with extras include:


		the ``extra1`` dependency would be included. If the user instead uses::
		the ``extra1`` dependency would be included. If the user instead uses:

	the ``extra1`` dependency would be included. If the user instead uses:
	the default ``extra1`` dependency would be included. If the user instead uses:


		pip install package package[extra2]
		$ pip install package package[extra2]

	Avoiding adding too many default dependencies
	Avoiding the addition of many default dependencies

PEP 771: Amendments following pre-publish discussion #4238

Are you sure you want to change the base?

PEP 771: Amendments following pre-publish discussion #4238

Conversation

astrofrog commented Jan 29, 2025 • edited by github-actions bot Loading

AA-Turner left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

astrofrog commented Jan 30, 2025

willingc left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

astrofrog commented Jan 29, 2025 •

edited by github-actions bot

Loading

AA-Turner left a comment •

edited

Loading

willingc left a comment •

edited

Loading