Script to Compare Packages in Benchpark Against Spack Tag

[michaelmckinsey1]

Description

• Add a new script to report whether packages in benchpark/repo are the same as var/spack/repos/builtin/packages.
• Addresses Check if package in repo matches package in Spack upstream #596
• Complete the checklist for a relevant section(s) below
• Delete sections below that are not relevant to this PR

Todo:

• superceded by Docs for developer scripts #695 -> We need a doc page for utilities in lib/scripts, perhaps call it "Benchpark Utils".
• Try running this script as CI for package x if repo/x/package.py is modified

Adding/modifying core functionality, CI, or documentation:

• Add benchpark/lib/scripts/spackdiff.py
• Update bin/benchpark-python, pyproject.toml, and lib/benchpark/cmd/unit_test.py to make interactive python optional during execution of bin/benchpark-python. It should be disabled while using this script.
• Rename altdiff.py to diffSpecs.py to match new script name.

Examples

Note: terminal output is colorized

We can check against spack tags like v0.23.1, or commits like 0b3b49b, which benchpark develop uses.
python spackdiff.py --spack-tag="0b3b49b"

Comparing benchpark packages to packages in spack 0b3b49b
Previous HEAD position was 99405e6a4d nlcglib: add v1.1.0 (#44580)
HEAD is now at 0b3b49b4e0 installer.py: handle external roots the same (#44917)
ad
    ad/package.py does not exist in spack
amg2023
     7 different lines
babelstream
     28 different lines
branson
     1 different lines
caliper
     95 different lines
cray-mpich
     67 different lines
cublas
    cublas/package.py does not exist in spack
cusolver
    cusolver/package.py does not exist in spack
enzyme
     33 different lines
essl
     34 different lines
genesis
     87 different lines
gromacs
     59 different lines
hipsycl
     58 different lines
hpcg
     55 different lines
hpl
     13 different lines
hypre
     208 different lines
kokkos
     207 different lines
kripke
     28 different lines
laghos
     22 different lines
lammps
     438 different lines
lapack-xl
    lapack-xl/package.py does not exist in spack
md-test
    md-test/package.py does not exist in spack
mfem
     604 different lines
osu-micro-benchmarks
     42 different lines
phloem
    phloem/package.py does not exist in spack
quicksilver
     9 different lines
qws
    qws/package.py does not exist in spack
raja-perf
    raja-perf/package.py does not exist in spack
remhos
     20 different lines
rocblas
     97 different lines
rocsolver
     67 different lines
saxpy
    saxpy/package.py does not exist in spack
smb
    smb/package.py does not exist in spack
spectrum-mpi
     81 different lines
stream
     33 different lines

See which lines are different for branson
python spackdiff.py --spack-tag="0b3b49b" --print-diff

branson
     1 different lines
--- spack branson
+++ benchpark branson
@@ -26,7 +26,6 @@
     )

     version("0.81", sha256="493f720904791f06b49ff48c17a681532c6a4d9fa59636522cf3f9700e77efe4")

     version("0.8", sha256="85ffee110f89be00c37798700508b66b0d15de1d98c54328b6d02a9eb2cf1cb8")

-

     depends_on("mpi@2:")

 

     depends_on(Boost.with_default_variants, when="@:0.81")

@@ -49,3 +48,4 @@
         install(join_path(self.build_directory, "BRANSON"), prefix.bin)

         install("LICENSE.md", prefix.doc)

         install("README.md", prefix.doc)

+        install_tree("inputs", prefix.inputs)

So the only difference between benchpark and spack branson package.py is the line install_tree("inputs", prefix.inputs) in benchpark.

[pearce8]

We need a doc page for utilities in lib/scripts, perhaps call it "Benchpark Utils".

In the doc, please add how to use the spackdiff. Also the altdiff. Although we need better names for these; how about:

• spackdiff should be diffPackages
• altdiff should be diffSpecs

Is there a way to have a more uniform way of using these?

[pearce8]

@michaelmckinsey1 address #695, get the latest changes from develop into #648, update docs as needed.

[michaelmckinsey1]

https://github.com/LLNL/benchpark/actions/runs/14095972151/job/39483687323

Example of how the CI will fail if one or more package.py files are the same as in spack. Deleting the benchpark package.py will fix the CI test. Spack package.py's were copied over for the example, changes to the files were subsequently reverted.