feat: `dnf` module #377

fiftydinar · 2024-12-22T13:53:07Z

Changes

Recipe format

Recipe format changed a bit, to satisfy the new feature of dnf flags.

Existing features code changes

Adding repos is now using dnf -y config-manager addrepo --from-repofile=$repo for repo URLs & repo files, while COPR repos are using native dnf -y copr enable user/project.
COPR repos are written in separate copr: array & in simpler user/project format in recipe, instead of long URL in repos: array.
rpm-ostree override remove --install=$pkg is replaced with 2 operations of dnf removal & then install.
Single operation of removal + installation of dnf doesn't currently exist, but it's not a breaking behavior to not have it.
rpm-ostree override replace is replaced with dnf -y distro-sync --refresh --repo $repo $packages, which is compatible with all repos.
Adding the repository along with replacement is disabled, so it must be done before replacement in repos: array if repo doesn't exist.

New features

dnf group removal & installation. Those are declared as group-remove & group-install array in recipe.
They run before the packages installation.
Option for additional flags when doing installation or replacement of packages.
- install-weak-dependencies: option for declaring if weak dependencies (such as Recommended) should be installed.
  It modifies the install & replace commands itself using --setopt=install_weak_deps=True/False flag, it doesn't modify the dnf config file. It defaults to true (reflecting dnf defaults).
- skip-unavailable-packages: option for skipping unavailable packages when not available in repositories, or when not available on the system in case of replacing. Passes --skip-unavailable flag. Defaults to false.
- skip-broken-packages: option for skipping the installation/replacement of broken packages. Passes --skip-broken flag. Defaults to false.
- allow-erasing-packages: option for erasing/removing the problematic packages during dependency problems. Passes --allowerasing flag. Defaults to false.
Option for additional flags when doing removal of packages.
- remove-unused-dependencies is true by default. If false is specified, it passes the --no-autoremove flag, which won't remove any unused dependencies during removal operation.

Issues to be closed

Fixes: #361
Fixes: #370
Fixes: #325
Fixes: #282

github-actions

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

shellcheck

⚠️ [shellcheck] _{reported by reviewdog 🐶}
Did you forget to close this double quoted string? SC1078

modules/modules/dnf/dnf.sh

Line 91 in cd91b05

CLASSIC_PKGS+=("${PKG}")

📝 [shellcheck] _{reported by reviewdog 🐶}
This is actually an end quote, but due to next char it looks suspect. SC1079

modules/modules/dnf/dnf.sh

Line 98 in cd91b05

echo "Installing: ${CLASSIC_PKGS[*]}"

🚫 [shellcheck] _{reported by reviewdog 🐶}
'(' is invalid here. Did you forget to escape it? SC1036

modules/modules/dnf/dnf.sh

Line 100 in cd91b05

echo "Installing package(s) directly from URL: ${HTTPS_PKGS[*]}"

🚫 [shellcheck] _{reported by reviewdog 🐶}
Expected 'fi' matching previously mentioned 'if'. SC1047

modules/modules/dnf/dnf.sh

Line 100 in cd91b05

echo "Installing package(s) directly from URL: ${HTTPS_PKGS[*]}"

🚫 [shellcheck] _{reported by reviewdog 🐶}
Expected 'fi'. Fix any mentioned problems and try again. SC1072

modules/modules/dnf/dnf.sh

Line 100 in cd91b05

echo "Installing package(s) directly from URL: ${HTTPS_PKGS[*]}"

modules/dnf/module.yml

github-actions · 2024-12-22T21:04:11Z

modules/rpm-ostree/rpm-ostree.sh

📝 [shellcheck] _{reported by reviewdog 🐶}
Possible misspelling: REPOS may not be assigned. Did you mean REPO? SC2153

modules/modules/rpm-ostree/rpm-ostree.sh

Line 8 in 1137c8b

if [[ ${#REPOS[@]} -gt 0 ]]; then

…d `--refresh` for install operation

certifiedfoolio · 2025-03-10T14:16:34Z

Hopefully we can add it around that timeframe, yeah. Thanks for the progress update.

certifiedfoolio · 2025-03-13T00:37:23Z

hey @fiftydinar , is it possible to have a separate module for dnf4? it would prove useful for working with certain images, in my case, centos-bootc:stream10. I did start on this myself, and the modified module is at https://github.com/cherry-os/cherryOS/tree/live/modules/dnf. It mostly contains minor syntax changes and a few links to external bash scripts that modify the output of some dnf4 commands into json that can be parsed properly, due to dnf4 not having the --json flag. Thanks!

certifiedfoolio · 2025-03-13T16:43:29Z

hey @fiftydinar , is it possible to have a separate module for dnf4? it would prove useful for working with certain images, in my case, centos-bootc:stream10. I did start on this myself, and the modified module is at https://github.com/cherry-os/cherryOS/tree/live/modules/dnf. It mostly contains minor syntax changes and a few links to external bash scripts that modify the output of some dnf4 commands into json that can be parsed properly, due to dnf4 not having --json. Thanks!

certifiedfoolio · 2025-03-13T16:44:16Z

So far my dnf4 module works, you just can't add repositories with it. There may be other issues I'm unaware of though.

fiftydinar · 2025-03-13T18:03:50Z

hey @fiftydinar , is it possible to have a separate module for dnf4?

Hi,

What's the status of dnf5 for centos & similar images which still utilize dnf4? Will it come soon?
I would prefer to officially only have dnf5 module, to make things simpler & avoid potential confusion of some users.

Another option would be to detect if dnf4 is used, then if it is, just change the necessary syntax in the module for the commands.

Adding your unofficial dnf4 module as a note in the README can also suffice I think.

However, I would like @gmpinder & @xynydev opinions on this.

certifiedfoolio · 2025-03-13T23:11:52Z

hey @fiftydinar , is it possible to have a separate module for dnf4?

Hi,

What's the status of dnf5 for centos & similar images which still utilize dnf4? Will it come soon? I would prefer to officially only have dnf5 module, to make things simpler & avoid potential confusion of some users.

Another option would be to detect if dnf4 is used, then if it is, just change the necessary syntax in the module for the commands.

Adding your unofficial dnf4 module as a note in the README can also suffice I think.

However, I would like @gmpinder & @xynydev opinions on this.

Hey,
'dnf5' isn't available at all on CentOS Stream or EPEL yet. It's why i'm proposing a seperate dnf4 module.
Due to CentOS Stream/EPEL's release cycle, it may be quite a while, as in years, before dnf5 arrives on Stream/EPEL repositories.

gmpinder · 2025-03-14T04:43:45Z

So far my dnf4 module works, you just can't add repositories with it. There may be other issues I'm unaware of though.

We'd be more than happy for you to contribute what you have. I don't think mixing the two in a single module would be the best option as you claim it doesn't operate the same.

certifiedfoolio · 2025-03-14T16:36:19Z

We'd be more than happy for you to contribute what you have. I don't think mixing the two in a single module would be the best option as you claim it doesn't operate the same.

It operates the same; the only code that's been changed is the addition of two bash scripts and syntax changes for dnf commands. The thing is it outputs a weird error when I try to add a repository using the modified module.

certifiedfoolio · 2025-03-14T16:40:05Z

@gmpinder This is the error I seem to be getting when using the below structure to add a repository file:
Structure:

type: dnf
source: local
repos:
   files:
      - filename.repo

Error:

[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04   × Cannot find column 'id'
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04      ╭─[/tmp/modules/dnf/dnf.nu:258:6]
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  257 │   let repo_info = try {
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  258 │     ^bash /tmp/modules/dnf/dnf-repolist.sh
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04      ·      ──┬─
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04      ·        ╰── value originates here
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  259 │   } catch {
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  260 │     exit 1
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  261 │   }
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  262 │     | from json
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  263 │     | get id
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  264 │     | par-each {|repo|
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  265 │       try {
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  266 │         ^bash /tmp/modules/dnf/dnf-repoinfo.sh $repo
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  267 │       } catch {
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  268 │         exit 1
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  269 │       }
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  270 │         | from json
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  271 │     }
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  272 │     | flatten
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  273 │ 
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  274 │   # Return the IDs of all repos that were added
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  275 │   let repo_ids = $repo_info
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  276 │     | filter {|repo|
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  277 │       $repo.repo_file_path in $repo_files
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  278 │     }
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  279 │     | get id
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04      ·           ─┬
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04      ·            ╰── cannot find column 'id'
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04  280 │ 
[23:35:20 g.i/c/stem-bourbon:latest] => #48 27.04      ╰────

gmpinder · 2025-03-14T17:25:11Z

@gmpinder This is the error I seem to be getting when using the below structure to add a repository file:
Structure:
type: dnf
source: local
repos:
   files:
      filename.repo

The files needs to be an array

certifiedfoolio · 2025-03-15T22:17:58Z

@gmpinder
Apologies for the formatting mistake, but that isn't the problem.
If i'm understanding the logs correctly, nu cant find the column "id" in the value originating from one of my scripts, and i've already verified that the output of the script is as accurate to that of dnf5's as possible. Of course, I may be wrong.

gmpinder · 2025-03-15T22:52:55Z

@gmpinder
Apologies for the formatting mistake, but that isn't the problem.
If i'm understanding the logs correctly, nu cant find the column "id" in the value originating from one of my scripts, and i've already verified that the output of the script is as accurate to that of dnf5's as possible. Of course, I may be wrong.

Your error looks nothing like the code we have in this PR. You're using your own local changes. Please use our discord server for the discussion instead as it isn't pertinent to these changes.

certifiedfoolio · 2025-03-16T02:19:03Z

Gotcha.

evanc577 · 2025-03-17T17:34:09Z

Can support be added for adding repos with baseurl and (optional) gpg key for repos that don't provide .repo links (e.g. terra)?

Something like this:

dnf5 -y install --repofrompath "$reponame,$repobaseurl" --repo "$reponame" "--setopt=${reponame}.gpgkey=${gpgkey}" "${packages[@]}"

Or with --nogpgcheck if no gpg key is provided.

gmpinder

I'm ready to have this merged if y'all are

gmpinder · 2025-03-19T01:48:20Z

Can support be added for adding repos with baseurl and (optional) gpg key for repos that don't provide .repo links (e.g. terra)?

Something like this:
dnf5 -y install --repofrompath "$reponame,$repobaseurl" --repo "$reponame" "--setopt=${reponame}.gpgkey=${gpgkey}" "${packages[@]}"
Or with --nogpgcheck if no gpg key is provided.

In this case, you would want to create your own .repo file under files/dnf/ of your repo and specify the name of the file under repos.files in your recipe.

xynydev · 2025-03-19T07:17:44Z

modules/dnf/README.md

+### Fix Optfix
+
+* Specify a list of packages to fix optfix issues in the `optfix` field
+
+Example:
+```yaml
+type: dnf
+optfix:
+  packages:
+    - package1
+    - package2
+```


There is no such thing as an "optfix fix". Optfix is the fix.

Suggested change

### Fix Optfix

* Specify a list of packages to fix optfix issues in the `optfix` field

Example:

```yaml

type: dnf

optfix:

packages:

- package1

- package2

```

* Optfix is a script used to work around problems with certain packages that install into `/opt/`

* These issues are caused by Fedora Atomic storing `/opt/` at the location `/var/opt/` by default, while `/var/` is only writeable on a live system

* The script works around these issues by moving the folder to `/usr/lib/opt/` and creating the proper symlinks at runtime

* Specify a list of folders inside `/opt/`

Example:

```yaml

type: dnf

optfix:

packages:

- foldername # ADD AN ACTUAL EXAMPLE FOLDER HERE

Excuse me for the potentially problematic formatting.

xynydev · 2025-03-19T07:33:00Z

modules/dnf/README.md

+Replacing the kernel with `dnf` module is not done cleanly & some remaints of old kernel will be present.  
+Please use `rpm-ostree` module for this purpose until this `dnf` behavior is fixed.


Was this fixed already in some images? I think I remember hearing about this.

xynydev · 2025-03-19T07:35:24Z

modules/dnf/README.md

+## Package Group Removal
+
+### Remove Packages Groups


Remove all of these sorts of double-titles from the whole document. This is not a sensible structure.

## Doing a Specific Thing ### Do The Thing

When there are multiple ###-level titles below the top-level title, that is fine. Might only need minor rewording.

xynydev · 2025-03-19T07:44:53Z

modules/dnf/README.md

+* Specify packages to install in the `install.packages` field
+* Use the `repo` parameter to specify a specific repository for installation
+* Use the `%OS_VERSION%` variable to automatically determine the operating system version
+* Use flags such as `skip-unavailable`, `install-weak-deps`, `skip-broken` and `allow-erasing` to customize package installation


I think there should be a section in the README dedicated to explaining what all the available flags are and what they mean. Language such as such as should be avoided here to avoid ambiguity. I would rather format this as

* You can specify the following flags with this operation * ...

xynydev · 2025-03-19T07:58:54Z

modules/dnf/module.yml

+  type: dnf
+  # Add repos, COPR repos, and repo keys
+  repos:
+    cleanup: false # (optional) Set to true to automatically remove added repos at the end of the module execution


# (optional) Set to true to automatically remove added repos at the end of the module execution
# Add a repo file from a URL

IMO these types of comments are not something that need to be in the example. All of this information should be in the documentation in the schema file, and that will be included on the documentation page on the website as well.

The example should be a fairly clean and readable configuration that one could mostly copy into their own recipe if they wished to. Flags should not be set to their default state manually, they should only be set if there is a specific reason to use the non-default state, with a comment explaining the reasoning.

certifiedfoolio · 2025-03-20T23:18:00Z

Starting to get strange errors with this module.
dnf5 isn't recognizing config-manager as an argument, judging by the errors im getting.

gmpinder · 2025-03-20T23:26:00Z

@xynydev guess that's what happens when I use AI to create the README.

gmpinder · 2025-03-20T23:29:56Z

Starting to get strange errors with this module.
dnf5 isn't recognizing config-manager as an argument, judging by the errors im getting.

The dnf plugin will need to be installed first for that to work. Maybe we should have a check for that and install it if you're trying to use it

certifiedfoolio · 2025-03-21T00:24:18Z

The dnf plugin will need to be installed first for that to work. Maybe we should have a check for that and install it if you're trying to use it
It wasn't installed by default? Huh.

fiftydinar added 2 commits December 22, 2024 14:52

feat: dnf module

fd4ad28

chore: Add support for native installation of repos

cd91b05

github-actions bot reviewed Dec 22, 2024

View reviewed changes

fiftydinar added 13 commits December 22, 2024 16:33

docs: Revise the note about %OS_VERSION% in repos

cf3d495

fix: missing quote in echo

2f392e7

fix: Get raw COPR repos array elements without JSON brackets

6d1047f

chore: Auto-confirm prompts (-y) when adding repos

9d13c87

fix: Accidental syntax error for done

1396a18

chore: Remove useless echo

b898af2

chore: Install & remove using dnf in single operation cases

6523979

docs: Add note that URL is not preferred for copr repos

97566b8

fix: /opt symlink issue & migrate removal + install to dnf

0bb79d0

chore: Document code some more

70c5766

docs: Fix note about using install & remove at the same time

e01ed40

chore: Add dnf to modules.json

7d4aff9

chore: Add support for replacing packages

a4e9ba4

gmpinder reviewed Dec 22, 2024

View reviewed changes

modules/dnf/module.yml Outdated Show resolved Hide resolved

fiftydinar added 3 commits December 22, 2024 21:41

chore: Add option to enable or disable installation of weak dependencies

7da24f5

docs: Style backtick fix

d495b6a

fix(rpm-ostree): Symlinking /opt/ multiple times caused an error

1137c8b

github-actions bot reviewed Dec 22, 2024

View reviewed changes

fiftydinar added 9 commits December 22, 2024 22:12

chore: Rename weak-dependencies to install-weak-dependencies & ad…

eb522e3

…d `--refresh` for install operation

chore: Change copr: user/project formt to COPR user/project

57f8b22

fix: Syntax error due to redundant fi

7a5bcea

fix: Restore logic about not removing any newlines to COPR repos

c7ae430

feat: Add removal & installation of RPM groups

b1a7c6f

docs: Style fix

b7c0206

chore: Make sorting order in typespec better

feb0d71

typespec: Minor leftover fix

fba3840

chore: Fix typespec replace definition

e27ba39

fiftydinar added 2 commits March 10, 2025 15:20

docs: Add example of old/new replacement format

0ac6938

Merge branch 'main' into dnf-module

8dd4e31

docs: Update module.yml to use old and new keys for replace

425a55e

gmpinder previously approved these changes Mar 19, 2025

View reviewed changes

xynydev reviewed Mar 19, 2025

View reviewed changes

docs: Clean up example snippets

6557643

gmpinder dismissed their stale review via 6557643 March 21, 2025 04:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: `dnf` module #377

feat: `dnf` module #377

fiftydinar commented Dec 22, 2024 •

edited

Loading

github-actions bot left a comment

github-actions bot Dec 22, 2024

certifiedfoolio commented Mar 10, 2025

certifiedfoolio commented Mar 13, 2025 •

edited

Loading

certifiedfoolio commented Mar 13, 2025

certifiedfoolio commented Mar 13, 2025

fiftydinar commented Mar 13, 2025 •

edited

Loading

certifiedfoolio commented Mar 13, 2025 •

edited

Loading

gmpinder commented Mar 14, 2025

certifiedfoolio commented Mar 14, 2025 •

edited

Loading

certifiedfoolio commented Mar 14, 2025 •

edited

Loading

gmpinder commented Mar 14, 2025 •

edited

Loading

certifiedfoolio commented Mar 15, 2025 •

edited

Loading

gmpinder commented Mar 15, 2025

certifiedfoolio commented Mar 16, 2025

evanc577 commented Mar 17, 2025

gmpinder left a comment

gmpinder commented Mar 19, 2025 •

edited

Loading

xynydev Mar 19, 2025

xynydev Mar 19, 2025

xynydev Mar 19, 2025

xynydev Mar 19, 2025

xynydev Mar 19, 2025

xynydev Mar 19, 2025

certifiedfoolio commented Mar 20, 2025

gmpinder commented Mar 20, 2025

gmpinder commented Mar 20, 2025

certifiedfoolio commented Mar 21, 2025

		Replacing the kernel with `dnf` module is not done cleanly & some remaints of old kernel will be present.
		Please use `rpm-ostree` module for this purpose until this `dnf` behavior is fixed.

feat: dnf module #377

Are you sure you want to change the base?

feat: dnf module #377

Conversation

fiftydinar commented Dec 22, 2024 • edited Loading

Changes

Recipe format

Existing features code changes

New features

Issues to be closed

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Dec 22, 2024

Choose a reason for hiding this comment

certifiedfoolio commented Mar 10, 2025

certifiedfoolio commented Mar 13, 2025 • edited Loading

certifiedfoolio commented Mar 13, 2025

certifiedfoolio commented Mar 13, 2025

fiftydinar commented Mar 13, 2025 • edited Loading

certifiedfoolio commented Mar 13, 2025 • edited Loading

gmpinder commented Mar 14, 2025

certifiedfoolio commented Mar 14, 2025 • edited Loading

certifiedfoolio commented Mar 14, 2025 • edited Loading

gmpinder commented Mar 14, 2025 • edited Loading

certifiedfoolio commented Mar 15, 2025 • edited Loading

gmpinder commented Mar 15, 2025

certifiedfoolio commented Mar 16, 2025

evanc577 commented Mar 17, 2025

gmpinder left a comment

Choose a reason for hiding this comment

gmpinder commented Mar 19, 2025 • edited Loading

xynydev Mar 19, 2025

Choose a reason for hiding this comment

xynydev Mar 19, 2025

Choose a reason for hiding this comment

xynydev Mar 19, 2025

Choose a reason for hiding this comment

xynydev Mar 19, 2025

Choose a reason for hiding this comment

xynydev Mar 19, 2025

Choose a reason for hiding this comment

xynydev Mar 19, 2025

Choose a reason for hiding this comment

certifiedfoolio commented Mar 20, 2025

gmpinder commented Mar 20, 2025

gmpinder commented Mar 20, 2025

certifiedfoolio commented Mar 21, 2025

feat: `dnf` module #377

feat: `dnf` module #377

fiftydinar commented Dec 22, 2024 •

edited

Loading

certifiedfoolio commented Mar 13, 2025 •

edited

Loading

fiftydinar commented Mar 13, 2025 •

edited

Loading

certifiedfoolio commented Mar 13, 2025 •

edited

Loading

certifiedfoolio commented Mar 14, 2025 •

edited

Loading

certifiedfoolio commented Mar 14, 2025 •

edited

Loading

gmpinder commented Mar 14, 2025 •

edited

Loading

certifiedfoolio commented Mar 15, 2025 •

edited

Loading

gmpinder commented Mar 19, 2025 •

edited

Loading