Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

treewide: cleanup; move packages to pkgs/by-name; format via nixfmt-rfc-style #325128

Closed
wants to merge 2 commits into from

Conversation

NotAShelf
Copy link
Member

@NotAShelf NotAShelf commented Jul 6, 2024

Description of changes

Currently going over #43716 to figure out licenses for packages without one, and performing maintenance (as in cleanup and formatting) on packages before I update and add license in a separate PR.

  • Formatting refers to formatting a package with nixfmt-rfc-style.
  • "Cleanup" refers to removing rec, with lib and other bad patterns.

This'll probably be large as there are lots of packages to go over.

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@NotAShelf NotAShelf changed the title treewide: move packages to pkgs/by-name; format via nixfmt-rfc-style treewide: cleanup; move packages to pkgs/by-name; format via nixfmt-rfc-style Jul 6, 2024
@ofborg ofborg bot requested a review from AndersonTorres July 6, 2024 22:38
@ofborg ofborg bot requested a review from pSub July 6, 2024 22:46
@NotAShelf NotAShelf closed this Jul 7, 2024
@NotAShelf NotAShelf reopened this Jul 7, 2024
@drupol
Copy link
Contributor

drupol commented Jul 7, 2024

What's the reason for closing this?

@NotAShelf
Copy link
Member Author

I was told in Matrix that such a PR would be discouraged not to step on others' toes.

@drupol
Copy link
Contributor

drupol commented Jul 7, 2024

Weird, that was somehow legit to me.

@NotAShelf
Copy link
Member Author

I am happy to continue with this PR, but I would like to avoid roadblocks down the road stemming from a lack of consensus. If I can get a few more positive responses to this, I'll re-open and proceed.

@drupol
Copy link
Contributor

drupol commented Jul 7, 2024

Can you please ping here the maintainer who advised not to do this?

@NotAShelf
Copy link
Member Author

That'd be @dali99 I believe.

@drupol
Copy link
Contributor

drupol commented Jul 7, 2024

@dali99 Could you please elaborate on this PR ?

The only issue I see here is the PR title, besides that, it looked legit to me.
I often do this kind of things myself, never had any issue with that.

@dali99
Copy link
Member

dali99 commented Jul 7, 2024

For the right context see

See https://matrix.to/#/!kjdutkOsheZdjqYmqp:nixos.org/$PilZWDCiopphZRNWhHWCQ20bulZOY0ynW6nK5tPLElg?via=lossy.network&via=matrix.org&via=nixos.dev

and

https://matrix.to/#/!kjdutkOsheZdjqYmqp:nixos.org/$MACs3PednHwhc7EIDHG4DYWV9OW0hl2yBPcEApH8CzQ?via=lossy.network&via=matrix.org&via=nixos.dev

I believe the "stepping on toes" part comes from https://matrix.to/#/!kjdutkOsheZdjqYmqp:nixos.org/$c64jRjBheEYGyhl1XXNsZ-wUGfaahFwRkK3MsIXXG-o?via=lossy.network&via=matrix.org&via=nixos.dev

The main points are:

  • Moving packages as part of a "general cleanup" treewide is not encouraged
  • There has been significant historical turmoil around automatic formatting. And we are not at the point where automatic formatting is being done on the whole tree. We should wait for that.
  • Even still, doing these things might be okay, but I believe that doing so should be up to the owners/maintainers of these packages. Not as part of a individual persons' desire to "clean up" derivations around the repository.

This PR's main motivation seems to be the by-name/formatting. And thus I think these things should not be encouraged yet. Especially not done unilaterally without the real maintainer's involvement.

This contributor is new and the advice was more general than necessarily for only this package

@NotAShelf
Copy link
Member Author

You are not wrong that formatting and migrating is my motivation, but its not the main motivation.

I am more interested and adding missing licenses, but I'd rather also not leave the packages in their "legacy" state. Some derivations also urgently need maintenance updates, which I'd rather get out of the way while I am already here.

@drupol
Copy link
Contributor

drupol commented Jul 7, 2024

I'm also driven by the will to fix issues when I see them, and I have my own personal rule. Here it is:

If when I edit a Nix file to fix a specific problem AND if I see other issues within this file, I fix them in the same PR, but in different commits.

Basically, every time I edit a Nix file in a PR, it should be totally valid and following the current best practices at the end of the PR.

@eclairevoyant
Copy link
Contributor

eclairevoyant commented Jul 7, 2024

Btw, there is no such thing as exclusive ownership in nixpkgs; this is a collaborative project. What being a maintainer entails is that you are making yourself available to fix issues with the package, handle updates, approve PRs, etc. and that you ideally use the package and have some knowledge of whether any change will improve a package or cause a problem with it.

This does not preclude making changes to packages you are not a maintainer for; many of our contributions come from non-maintainers.

@NotAShelf, please feel free to reopen if you're interested in continuing work on this PR.

@drupol
Copy link
Contributor

drupol commented Jul 7, 2024

Btw, there is no such thing as exclusive ownership in nixpkgs; this is a collaborative project. What being a maintainer entails is that you are making yourself available to fix issues with the package, handle updates, approve PRs, etc. and that you ideally use the package and have some knowledge of whether any change will improve a package or cause a problem with it.

This does not preclude making changes to packages you are not a maintainer for; many of our contributions come from non-maintainers.

@NotAShelf, please feel free to reopen if you're interested in continuing work on this PR.

Voila quelqu'un d'eclairevoyant! (french pun intended!)

Please forgive me, it's been a while I wanted to do it...

@NotAShelf NotAShelf reopened this Jul 7, 2024
Copy link
Contributor

@drupol drupol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please adjust the PR title?

@NotAShelf
Copy link
Member Author

Sure, how should I adjust it?

@drupol
Copy link
Contributor

drupol commented Jul 7, 2024

Since you're modifying only one derivation, I would do:

cue2pops: move to `pkgs/by-name`

@NotAShelf
Copy link
Member Author

I plan to move other derivations as well, though. So far I have 7 waiting to be pushed.

@drupol
Copy link
Contributor

drupol commented Jul 7, 2024

Ooh fair enough. Ignore my comment then.

@dali99
Copy link
Member

dali99 commented Jul 8, 2024

Btw, there is no such thing as exclusive ownership in nixpkgs; this is a collaborative project. What being a maintainer entails is that you are making yourself available to fix issues with the package, handle updates, approve PRs, etc. and that you ideally use the package and have some knowledge of whether any change will improve a package or cause a problem with it.

This does not preclude making changes to packages you are not a maintainer for; many of our contributions come from non-maintainers.

It was never about exclusive ownership. I'm arguing for respecting each other's responsibilities and following the rules, guidelines, and plans we've agreed on. These are both crucial parts of a collaborative project. I'm just accepting there's nuance around ignoring the plan for the things you're actively responsible for.

Misconstruing my sentiment to imply anything else is - how should I say this - Totally uncool.

Basically, every time I edit a Nix file in a PR, it should be totally valid and following the current best practices at the end of the PR.

My own ethos is:
Fix problems, keep the local style, make it easy to accept. As if they wrote it.
Be non-controversial

@eclairevoyant
Copy link
Contributor

What rule is this PR violating, exactly?

@emilazy
Copy link
Member

emilazy commented Jul 8, 2024

I think formatting files you touch is in the spirit of #322537, which will soon make it mandatory precisely to enable this kind of incremental transition; there’s clearly no question of ownership there and no real grounds to hold back on formatting any given leaf package that doesn’t have a ton of existing open PRs.

Moving into pkgs/by-name I can understand there being some maintainer discretion on; the story for doing that across the entire tree hasn’t been entirely worked out there, and CI ratchet checks means that there’s no easy way of undoing it in case some of the functionality that cannot be used with pkgs/by-name becomes required at a later date. Still, for simple leaf packages it also seems fine to me.

If a maintainer objects then of course their reasons should be taken into consideration, and I do think that those migrations should be in separate commits from any other changes to make reviewing the diffs easier. But, while I grant that there is some nuance here, I personally disagree with @dali99 about the extent to which there is any firm consensus or guideline against doing these clean‐ups, as long as you have another meaningful change to make to the packages anyway. The documentation for pkgs/by-name states:

Manual migrations should only be lightly encouraged if the relevant code is being worked on anyways. For example with a package update or refactoring.

…which, while hardly an urgent endorsement, certainly doesn’t advise against it.

@AndersonTorres
Copy link
Member

ofborg called me to review this, since the package I maintain was touched by it.

Let's go:

  1. Mere migration to by-name is OK to me. I would do this myself, however this was discouraged.
  2. Migration plus modernization (remove nested with) is OK too.
  3. nixfmt-rfc-style I am a bit reticent, since it can generate noisy patches.
    Further, many people hate to review PRs with reformatting.

Some derivations also urgently need maintenance updates, which I'd rather get out of the way while I am already here.

This should be made on a per package basis, not as part of a treewide PR.

@emilazy
Copy link
Member

emilazy commented Jul 8, 2024

If every minor clean‐up has to be split into one PR per package, people just won’t clean things up. I understood the intent of this PR to be to handle the packages listed as lacking licence information in #43716; I think doing some tidying up along the way is reasonable. I do think there’s a threshold of change where it makes sense to split things up and get individual review but I’m not sure that would reach it.

That said, I see that the change currently in the PR is just a mechanical update and clean‐up of a package that isn't lacking licence information; for that kind of thing I agree that it should be either separated out into its own PR or done systematically in one go.

@drupol
Copy link
Contributor

drupol commented Jul 8, 2024

The more people we involve, the more we will have different opinionated overviews.
I'm here to improve nixpkgs, not for those politics and shenanigans, unsubscribing.

@dali99
Copy link
Member

dali99 commented Jul 8, 2024

My personal opinion is that we should have done treewide formats and by-name moves a long time ago. And that at this point the churn generated by not doing this is worse than any merge conflicts we would've had. But still I'm arguing for following the plan.

If a maintainer objects then of course their reasons should be taken into consideration

When we don't have consensus it's more important to be careful. Not having consensus doesn't mean you can do whatever you prefer - it means you should tread lightly. Especially for treewides

Social dynamics are complex, this is a place where it can be hard to say no or to object to things. Therefore a large part of the responsibility lies with each of us to try to avoid putting each other in these situations in the first place.

What I mean by this more concretely is that you shouldn't put the responsibility on other people to object.
We're not in a hurry here, both of these things will take care of themselves with time.

Don't make people feel as though they've been ran over just because they're quiet.

@NotAShelf NotAShelf closed this Jul 8, 2024
@emilazy
Copy link
Member

emilazy commented Jul 8, 2024

I think there is perhaps some confusion here because I believe “treewide:” in the PR is only meant to reflect that it touches multiple packages rather than an intent to touch every single package. But I don’t feel like I’m doing a good job at conveying the nuances of my position here, so I’ll bow out too.

@dali99
Copy link
Member

dali99 commented Jul 8, 2024

@emilazy I feel you. I originally spent hours yesterday trying to convey my feelings as accurate as possible. Culminating in a very large wall of text and one light mental breakdown about the changing culture in nixpkgs over the last six years (suffice to say I didn't send it). I felt quite hurt by @eclairevoyant s comment. But I'm not good at writing accurately, concisely, and politely.

I'm sure we all actually have much more nuanced opinions than what we are managing to express here.

@NotAShelf I'm really sorry your PR was turned into this

@eclairevoyant
Copy link
Contributor

When we don't have consensus

NixOS/rfcs#140 (by-name structure) and NixOS/rfcs#166 (nixfmt-rfc-style) getting merged represents net community consensus. There is nothing to tread lightly here about, as this follows the guidelines/plans/whatever as written in this very repo. @emilazy quoted the relevant guideline for the former; for the latter, we already have CI enforcing the formatter on certain paths:

https://github.com/nixos/nixpkgs/blob/d14441cfe1bba01699cd614d6ef82cf6bc6aac22/.github/workflows/check-nix-format.yml#L37-L56

So I'm not sure why you think there is a lack of consensus here.

@AndersonTorres
Copy link
Member

If every minor clean‐up has to be split into one PR per package

I am not talking about every minor cleanup.

I am talking about "maintenance updates".

"Maintenance updates" has a large latitude of meaning.

I am doing a maintenance update on the SDL ecosystem, since the original maintainers retired.
I am not introducing any new package. I am merely reworking the code.
And it is causing mass rebuilds:

I tried to do maintenance updates to mesa expressions, and a single whitespace after a -D caused mass rebuilds.

Maintenance updates are not necessarily minor cleanups.

I think doing some tidying up along the way is reasonable.

I agree with you here.
However, there are many people - and among them, many people with commit bit - that don't.
And they can easily embargo a PR.

@NotAShelf NotAShelf deleted the treewide-by-name-fmt branch July 8, 2024 14:26
@dali99
Copy link
Member

dali99 commented Jul 8, 2024

When we don't have consensus

NixOS/rfcs#140 (by-name structure) and NixOS/rfcs#166 (nixfmt-rfc-style) getting merged represents net community consensus. There is nothing to tread lightly here about, as this follows the guidelines/plans/whatever as written in this very repo. @emilazy quoted the relevant guideline for the former; for the latter, we already have CI enforcing the formatter on certain paths:

nixos/nixpkgs@d14441c/.github/workflows/check-nix-format.yml#L37-L56

So I'm not sure why you think there is a lack of consensus here.

I have to believe you are being intentionally obtuse here and I will therefore disengage, this is the second time you've managed to carefully misunderstand my entire argument. This is entirely a question of how, when and not if, and has always been this. The fact that we've agreed on that this will happen does not mean we've agreed on it happening this way, or that it will happen now. There was discussion around enabling the formatting lint in for example python. But they can do that because the python team is responsible for the python tree.

We obviously disagree as to whether or not this fits the "exception" for by-name migrations. But the main crux of the argument isn't that a package can or can't be moved. It's that you might not the one who should be making that decision. I don't believe this light encouragement trumps giving that agency to our maintainers. You might believe something different.

@eclairevoyant
Copy link
Contributor

eclairevoyant commented Jul 8, 2024

I'm not misunderstanding anything; you seem to believe that maintainership is a form of ownership of a package and they have the final say on every change that happens on the package and can even block improvements, even if they don't actually cause problems to the packaging. I cannot agree with that, nor have I seen that being put into practice.

Especially when we have ratcheting checks that will eventually force the move and reformatting anyway, discouraging/bikeshedding someone that you refer to as a "new contributor" simply doing the sort of PR that has been done hundreds of times already seems unnecessarily hostile and capricious.

Though I can see a good argument for breaking this up into multiple PRs instead of a treewide, as suggested above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants