Suggestions for using list_url and friends

[knoepfel]

We have many package recipes that contain blurbs like:

class Nutools(CMakePackage):
    homepage = "https://cdcvs.fnal.gov/redmine/projects/nutools/wiki"
    git = "https://github.com/NuSoftHEP/nutools.git"
    url = "https://github.com/NuSoftHEP/nutools/archive/refs/tags/v3_11_05.tar.gz"
    list_url = "https://api.github.com/repos/NuSoftHEP/nutools/tags"

    version("3.15.04", sha256="9f145338854ae1bbcfbbbd7f56fd518663cfd0e2279520c31649ad1b71d4d028")
    ...
    version("develop", branch="develop", get_full_repo=True)

    def url_for_version(self, version):
        url = "https://github.com/NuSoftHEP/{0}/archive/v{1}.tar.gz"
        return url.format(self.name, version.underscored)

    def fetch_remote_versions(self, concurrency=None):
        return dict(
            map(
                lambda v: (v.dotted, self.url_for_version(v)),
                [
                    Version(d["name"][1:])
                    for d in sjson.load(
                        spack.util.web.read_from_url(
                            self.list_url, accept_content_type="application/json"
                        )[2]
                    )
                    if d["name"].startswith("v")
                ],
            )
        )

I will start by explaining the behavior of the various components, and how they work together. I will then propose an adjustment to how we do things.

Our fetch_remote_version override

The fetch_remote_version method is used by Spack to get the list of available versions when invoking spack checksum <package>. The package_base class provides a default implementation, but we often override it with an implementation that fetches versions directly from self.list_url (with a bit of filtering).

list_url

list_url is an attribute that Spack uses in its own implementation of fetch_remove_version to expand the set of URLs to search for versions. When we provide our own fetch_remote_version method, Spack no longer directly uses list_url for anything (as far as I can tell).

url

By default, Spack uses the version pattern specified in url to determine how it should look for versions in the various URLs it spiders. For example, a specification of:

url = "https://github.com/NuSoftHEP/nutools/archive/refs/tags/v3_11_05.tar.gz"

tells Spack to search for versions of the form vX_YY_ZZ in not only the URL as specified on the line, but the list_url as well, and any directories the spidering mechanism can access (including subdirectories of the landing page).

Of course, since we often provide explicit url_for_version and fetch_remote_version overrides, the url is likely only used when printing package information via spack info <package>.

Problems with our current mechanism:

• Incurred technical debt using our own implementation(s) of fetch_remote_version
• The output from a list_url of (e.g.) https://api.github.com/repos/NuSoftHEP/nutools/tags is paginated, keeping only 30 versions unless the ?per_page=<some number bigger than 30> suffix is added to the URL.

The suggestion

The above blurb should be replaced with:

from spack.pkg.fnal_art.fnal_github_package import *


class Nutools(CMakePackage, FnalGithubPackage):
    """Nutools"""

    repo = "NuSoftHEP/nutools"
    license("Apache-2.0")
    version_patterns = ["v3_15_04", "3.16.03"]

    version("3.16.05", sha256="030cad7d6b7d8c079543203ef5c60292c961d5cc8f6acfd16e360209adda6a0c")
    version("3.15.04", sha256="9f145338854ae1bbcfbbbd7f56fd518663cfd0e2279520c31649ad1b71d4d028")
    version("develop", branch="develop", get_full_repo=True)

where version_patterns follows the following convention:

• All versions from 3.15.04 up to (but excluding) 3.16.03 have a tarball version string of the form .../vX_YY_ZZ.tar.gz
• All versions from 3.16.03 and above have a tarball version string of the form .../X.YY.ZZ.tar.gz

The class FnalGithubPackage is a mixin class provided through the fnal-github-package Spack package from the fnal_art repository. Under the covers, FnalGitHubPackage, provides urls and list_url attributes as well as a homepage attribute if one is not already provided by the package itself.

Consequences of this suggestion

• No more custom implementations of fetch_remote_version.
• All versions following the pattern X.YY.ZZ will automatically be recognized from https://github.com/NuSoftHEP/nutools/tags.
• Versions following the pattern vX_YY_ZZ will no longer be automatically listed during spack checksum <package>, but if a particular version is desired spack checksum <package> <version> with an explicit version will still work due to the url_for_version definition. I think this is okay because going forward, we are getting away from the UPS-style version strings.

If this suggestion is approved, tags will need to be generated that follow the dot-delimited pattern, and not the UPS-style one. We've started to do this with the nutools packages (and perhaps others).