Skip to content

add hook that copies the easybuild subdirectory of every installation to a central reprod directory #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

bedroge
Copy link
Contributor

@bedroge bedroge commented Jun 26, 2025

This implements two hooks that will improve the reproducibility of a stack: one that copies missing patches from extensions to $installdir/easybuild, and one that copies the entire $installdir/easybuild to stack_root/reprod/app_name/app_version/timestamp/.

See https://gitlab.com/eessi/support/-/issues/145#note_2437292887 for more details.

Still WIP, also because the post_easyblock_hook is not available yet and will be part of the next release (EB 5.1.1).

@bedroge bedroge marked this pull request as draft June 26, 2025 15:23
…only copy easybuild subdir with post_easyblock_hook
@bedroge
Copy link
Contributor Author

bedroge commented Jun 27, 2025

This should work now, but it needs a new EB release as it depends on easybuilders/easybuild-framework#4938 and easybuilders/easybuild-framework#4939.

@bedroge bedroge changed the title WIP: hooks for improving reproducibility add hook that copies the easybuild subdirectory of every installation to a central reprod directory Jul 7, 2025
@bedroge bedroge marked this pull request as ready for review July 7, 2025 09:48
@bedroge
Copy link
Contributor Author

bedroge commented Jul 7, 2025

Still needs EESSI/software-layer#1135 to be ingested first.

Copy link
Member

@ocaisa ocaisa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a tiny change needed

Co-authored-by: ocaisa <[email protected]>
@ocaisa
Copy link
Member

ocaisa commented Jul 8, 2025

@bedroge Can you add a temporary test build here to make sure this is working as expected?

@bedroge
Copy link
Contributor Author

bedroge commented Jul 8, 2025

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-jsc architecture:aarch64/nvidia/grace

@eessi-bot-jsc
Copy link

eessi-bot-jsc bot commented Jul 8, 2025

New job on instance eessi-bot-jsc for CPU micro-architecture aarch64-nvidia-grace for repository eessi.io-2023.06-software in job dir /p/project1/ceasybuilders/eessibot/jobs/2025.07/pr_24/13867218

date job status comment
Jul 08 15:49:54 UTC 2025 submitted job id 13867218 awaits release by job manager
Jul 08 15:50:27 UTC 2025 released job awaits launch by Slurm scheduler
Jul 08 15:51:32 UTC 2025 running job 13867218 is running
Jul 08 15:54:45 UTC 2025 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-13867218.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-aarch64-nvidia-grace-17519899910.tar.gzsize: 0 MiB (90034 bytes)
entries: 80
modules under 2023.06/software/linux/aarch64/nvidia/grace/modules/all
cowsay/3.04.lua
software under 2023.06/software/linux/aarch64/nvidia/grace/software
cowsay/3.04
other under 2023.06/software/linux/aarch64/nvidia/grace
2023.06/init/easybuild/eb_hooks.py
Jul 08 15:54:45 UTC 2025 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-13867218.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@bedroge
Copy link
Contributor Author

bedroge commented Jul 8, 2025

@bedroge Can you add a temporary test build here to make sure this is working as expected?

Done for 2023.06. We can't test it for 2025.06 yet, as we first need to have some scripts and a module file in place.

@bedroge
Copy link
Contributor Author

bedroge commented Jul 8, 2025

If you'd really like to test 2025.06 as well, we should first deploy+merge #30.

@ocaisa
Copy link
Member

ocaisa commented Jul 8, 2025

It doesn't seem to work? No reprod directory in the tarball...

@bedroge
Copy link
Contributor Author

bedroge commented Jul 8, 2025

It doesn't seem to work? No reprod directory in the tarball...

I've disabled the hook for EESSI version 2023.06, as I didn't see much value in only doing that for new builds (we will have an incomplete reprod dir). On the other hand, it doesn't do any harm, so we could enable it?

(edit: the dummy build was just to prove that it doesn't break anything for 2023.06).

@ocaisa
Copy link
Member

ocaisa commented Jul 8, 2025

Argh, sorry, wasn't paying enough attention

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants