What is a 'package' ?

[JimFuller-RedHat]

No where in the spec (or README) do we explicitly define what a package is ... of course we might infer:

 'A package is a software component(s) installed by a package manager.'

If this is enough, perhaps we define the term in the spec ?

I see the sense of defining package management as a boundary that makes things tractable though it might be useful to provide advice for how to handle things outside of package management. Interested in others thoughts...

[di]

Duplicate of #257?

[JimFuller-RedHat]

thx for finding this ! overlaps and fleshes out some of the implications - but does not answer the question ;)

Putting on my spec lawyer hat - a lot of that issue revolves around a pURL being a URL ... we seem to intertwingle the notion of using a URL as a locator (and all the vagaries of dereferencing) versus a pURL as an opaque identifier. In an alternate universe I think they might be using URI for plain ole identification of software components ;)

[pombredanne]

@JimFuller-RedHat I would not want to wake up the spec lawyer in you, as this feels intimidating 😊 ! ... now to me, a PURL is a URL to locate, and also to identify.

So, what's a package? I like to think as the package "type" or ecosystem as defining a package "protocol", i.e., a number of conventions, formats, network protocols, and specs that help us name, locate, fetch, reuse, and interact with code packaged for this purpose by that ecosystem.

As an example for Python, this encapsulate a large number of things that collectively form the "pypi" "protocol", eventually documented in multiple PEPs, PyPA documents, down to unwritten tribal knowledge:

• various metadata and manifests formats used to describe and build a package (setup.py/.cfg/pyproject.toml) and their specs
• packaging formats (wheels, tar balls for sources) and their metadata and naming conventions
• the various tools that can install and build packages (pip, build, poetry, hatch and so on) and the libraries that process the formats and metadata (such as packaging, etc.)
• the pypi.org main site and its API, including the "simple" API, documented and spec'ed as the reference and the way to get download URLs and verify the content with checksums, and so on.
• the extra pypi.org-provided metadata such as download stats and vulnerabilities
• the pysec security database
• the metadata URLs that link towards the rest of the world, source repos, issues and so on

So here "pypi" means all this, and a package is in this context would be:

• collectively, all the wheels and sdist for a given name@version
• specifically, a given wheel or tarball for a given name@version (with some qualifier)

The same reasoning would extend to other types.
For instance, is a git repo a package?

• I would say no, this is at best a URL to a version control system. Why? because it lacks many of the "protocol" characteristics as with "pypi"

And is a github or gitlab repo a package?

• I would say yes, using github or gitlab as a type determines a host of metadata, conventions and APIs that make these rise to the level of a "package"

And of course, because all abstractions eventually leak, we have an escape hatch with a "generic" type and a few other attempts, but these are really hatches and best used sparingly.

I hope this helps. We could and should draft some intro blurb for the spec, as the id vs. locate and the "what's a package" question are things that come up often.

[JimFuller-RedHat]

Ah sorry not trying to instill anxiety ;) ... pURL has done a fine job avoiding spec churn (and I certainly do not want to encourage any with 'rabbit hole' conversations on esoterica ) ... I do think it is worthwhile to come up with a stable definition of what a pURL describes.

A pURL is used to identify a software package (with the 'boundary' of software package as the subject of this discussion) - I am not aware of any parsers, libraries or apps directly de-refencing the following:

  pkg:bitbucket/birkenfeld/pygments-main@244fd47e07d1014f0aed9c

Parsing most pURL with trurl fails because it works only with hierarchical urls: ... though if we add the notorious '//' (ignore this, trurl prob needs to add a flag for this!!):

trurl --url pkg://bitbucket/birkenfeld/pygments-main@244fd47e07d1014f0aed9c --json
[
  {
    "url": "pkg://bitbucket/birkenfeld/pygments-main%40244fd47e07d1014f0aed9c",
    "parts": {
      "scheme": "pkg",
      "host": "bitbucket",
      "path": "/birkenfeld/pygments-main@244fd47e07d1014f0aed9c"
    }
  }
]

another more valid example:

trurl --url "pkg://rpm/redhat/[email protected]?arch=i386" --json
[
  {
    "url": "pkg://rpm/redhat/curl%407.50.3-1?arch=i386",
    "parts": {
      "scheme": "pkg",
      "host": "rpm",
      "path": "/redhat/[email protected]",
      "query": "arch=i386"
    },
    "params": [
      {
        "key": "arch",
        "value": "i386"
      }
    ]
  }
]

I use trurl as the gold standard for what a valid URL is - once you start digging into URLs they become hideously complex (looking at you WHATWG) ... ex. pURL pushes complexity of utf8 and IRI aside in a faustian exchange for rules on percent encoding. So using trurl now is not exactly right, but when we get an IANA registry entry and we investigate semantics we might find some of the current rules of pURL invalidates it being a true URL (in the URL valid sense). Why the pedantry, well because it may have implications for developers implementing tooling that uses, parses, generates pURL.

To be able to be used as a locator the pkg: scheme probably need to have some defined protocol semantics ... though we would hope a pURL, armed with knowledge of type (and implied protocol), identifies a software package 'coordinates' to be able to de-reference from somewhere but in reality (and in the wild) we have a mixture of incomplete and concrete pURLs, for example:

    pkg:rpm/fedora/curl
    pkg:rpm/fedora/[email protected]
    pkg:rpm/fedora/[email protected]?arch=i386
    pkg:rpm/fedora/[email protected]?arch=i386&distro=fedora-25
    pkg:rpm/redhat/[email protected]?arch=i386

What do any of the above 'mean' ? How would a consumer know when a pURL is referring to a single 'bag o bits' ? A URL can of course represent a list of resources or a single resource - such ambiguity in the spec can be a good thing but might be worth explicitly defining (like saying its up to the package ecosystem to define its meaning).

For example - the above pURLs might be equivalent and predicate a notion of aliases (the world of HTTP offers 'food for thought' with their HTTP redirect semantics, url shorteners and more) ... I do not think the pURL spec has to define aliasing but we should try to draw a boundary around what a pURL describes to contain it. In this instance it might be useful to explicitly state in the spec what we want it to mean (ex. a single software package, a set of software package, both, etc ...).

The spec defines all pURLs MUST use pkg: scheme (which needs an IANA registry entry) and then mentions about not being any special schemes or other schemes (those bullet points seem superfluous and could be removed).

The definition of pkg: scheme implies that most semantics are contextually dependent on pURL types ... though that document does not define if any qualifiers are required/optional.

---

The definition of a software package as an entity that participates in a system with an algebraic set of operations on data defining a package protocol which is mainly concerned with distribution (rewrote your succinct definition!) seems reasonable though it implies:

If software package identity is a 'side effect' of the software component participation in a package ecosystem then I (maybe wrongly) interpret this to mean that a software package does not yet 'exist' until it is a member within an ecosystem.

In the past, we may have opted to identify a software component based on build characteristics, for example I could build curl with:

./configure --prefix=/usr/local --with-openssl --enable-http2 --enable-shared --with-zlib

and we might propagate these as qualifiers in some identity strategy ... there is still ambiguity in that at runtime exactly which code paths get 'lit up' is heavily dependent on execution env context. In maven world this can be even more complicated eg. reviewing what is running in the JVM at runtime vs defined at build vs install time. The point is that this kind of information (we hope) is indirectly introspected via relationship of pURL to package ecosystem though identifying a concrete component at runtime seems outside of scope (as one does not consult a package manager to tell you about runtime identity of a software component).

It seems like we cannot reliably mint pURL on anything outside that definition - how to identify software before we formalise in a package distribution system ... or how to identify software that has deprecated/expired and out of distribution ? There is software that has no package management but still needs identity... perhaps we should explicitly state such things are out of scope for pURL ?

In the wild, we ab/use generic pURL type - which invalidates the package ecosystem definition of a software package ... if we define a pURL as identifying a 'things' existence within a package ecosystem, then define an 'escape hatch' which allows for defining everything ... not sure about that.

So how do we identify software:

• Before package management: All software starts somewhere aka build time .. security domain in particular needs to 'shift earlier' to catch problems sooner.
• Software outside of package management: This is more about software that will never have package management ... perhaps we just say out of scope (which does not solve the problem).
• After package management: We are probably less interested in this scenario though it raises interesting questions that a software package that is no longer distributed can still have a pURL.
• Runtime identity: Runtime seems to be implied out of scope. We could assume there is a strong relationship between an installed package and running software component(s) though 'all bets are off' (esp in some ecosystems).

---

Summary: if we use the following definition:

 'A package is a software component(s) installed and managed by a package ecosystem (as defined in pURL types).'

Then we might want to address some of the points I raise above. I am also hopeful we could do better then the generic type 'band aid'. If pURL as locator is a goal we need to tighten up type definitions though not so convinced that is as important as its usage as a unique identifier. Untangling what (and if) pURL describes at build, package/distribute and runtime is a challenge.

[pombredanne]

@JimFuller-RedHat you wrote:

I am not aware of any parsers, libraries or apps directly de-refencing the following:

  pkg:bitbucket/birkenfeld/pygments-main@244fd47e07d1014f0aed9c

Argh! The law of unintended consequences is catching up here!

• pkg:bitbucket/birkenfeld/pygments-main@244fd47e07d1014f0aed9c used to be a valid PURL and back then this was hosted on mercurial.
• https://bitbucket.org/birkenfeld/pygments-main is now dead and the project migrated (without keeping a page up, may be because of some IMHO quite unfortunate bitbucket goofing things up when they stopped supporting mercurial and just deleted the old repos? not sure)
• there is some old archive at https://github.com/pygments/pygments-archive
• pygments now lives at https://bitbucket.org/birkenfeld/pygments-main
• 244fd47e07d1014f0aed9c is a commit that is nowhere to be found on GitHub or in git repositories, since it was likely a mercurail commit and I cannot find a mirror of the older hg code with commits ids.
  • pygments/pygments@244fd47e07d1014f0aed9c is of course 404

The unintended consequences are that since I used that example in the spec, this is now seen in many places.... See these links below, with specs, libraries and docs reference that dead commit from US NTIA docs, OMG CSAF spec, SPDX spec, most of the purl libraries all reference this mostly dead PURL as a valid one 😿 😿 😿 😿 :

• https://www.google.com/search?q="244fd47e07d1014f0aed9c"&pws=0&gl=us&gws_rd=cr
• https://github.com/search?q=244fd47e07d1014f0aed9c&type=code

Thanks for catching that one up. So de-referencing, as in "redirecting to the new GitHub repo" with a new PURL would be difficult here: commits have not the same id, hg history is missing, and there is little explicit information about the move on that would help reconstruct this.

sigh ... I guess this could mean that:

• we should track old PURLs for proper processing at scale
• we should archive all the code, all the time

SWH may help a bit there https://archive.softwareheritage.org/browse/search/?q=pygments-main&with_visit=true&with_content=true but this should be a last resort.

[JimFuller-RedHat]

I choose the example to serve 'double duty' and as an opportunity to refer to https://www.w3.org/Provider/Style/URI ;) (like me it is old, dated but hopefully still useful)... an identifying scheme should be stateless (to scale and be distributed) and not depend on things like 'tracking old pURLs' ...

[matt-phylum]

The unintended consequences are that since I used that example in the spec, this is now seen in many places.... See these links below, with specs, libraries and docs reference that dead commit from US NTIA docs, OMG CSAF spec, SPDX spec, most of the purl libraries all reference this mostly dead PURL as a valid one 😿 😿 😿 😿 :

I think this is an unavoidable reality that's true with URLs as well. I deal with a lot of PURLs that can no longer be resolved to packages because package registries normally delete malicious packages. At that point you only have the meaning encoded into the PURL, which isn't very informative for pkg:bitbucket.

[JimFuller-RedHat]

yes everything dies, and that is a good thing ... the problem however with identity is that naming new things often can take old names ... for (a real meta) example there is a PURL specification which is not pURL ... a full URL should disambiguate - just want to to make sure we think through implications - another example, is a purl a purl if it locates nothing anymore (in a package ecosystem) ?