DESCRIPTION version constraints with upper and lower bounds

[tannerntannern]

I'm using renv::install() command to install constrained package versions from a DESCRIPTION file, as described by https://rstudio.github.io/renv/articles/packages.html

Annoyingly, it doesn't seem possible to put an upper bound on a package version if that upper bound doesn't exist. For example, I'd like to use the following in my DESCRIPTION file:

Imports:
    dplyr (>= 1.0.0, < 1.1.0)

In other words, I'm ok with any patch releases of dplyr, but no minor releases. However, renv::install() doesn't seem to like this because 1.1.0 doesn't exist (yet):

> renv::install()
Error: package '1.1.0' is not available
Traceback (most recent calls last):
7: renv::install()
6: retrieve(names(remotes))
5: handler(package, renv_retrieve_impl(package))
4: renv_retrieve_impl(package)
3: renv_available_packages_latest(record$Package)
2: stopf("package '%s' is not available", package)
1: stop(sprintf(fmt, ...), call. = call.)

I could alternatively set the upper bound to the latest released version of dplyr, but that won't necessarily include future patch releases. Is there a technical reason why the upper bound must exist for the command to succeed? This wasn't a limitation I was expecting having used package managers for other languages.

Thanks in advance for your time!

[tannerntannern]

Hmmm, after some more trial and error it seems like this syntax doesn't work at all with renv. When I tried with dplyr (>= 1.0.2, < 1.0.4) (where both package versions definitely exist), I get the same error:

> renv::install()
Error: package '1.0.4' is not available
Traceback (most recent calls last):
7: renv::install()
6: retrieve(names(remotes))
5: handler(package, renv_retrieve_impl(package))
4: renv_retrieve_impl(package)
3: renv_available_packages_latest(record$Package)
2: stopf("package '%s' is not available", package)
1: stop(sprintf(fmt, ...), call. = call.)

Can someone confirm or deny that this type of constraint is even supported in the first place? I'm finding it difficult to get a definitive answer on this in my online searches.

[tannerntannern]

Closing in favor of #767

[kevinushey]

It is not supported; R and renv only support a single constraint on the package version.

Note that renv does not actually have a solver like you are probably imagining it does -- in general, it knows how to either:

1. Install the latest-available version(s) of some packages, or
2. Install the exact requested versions.

See pak if you want something more like an R package installer that attempts to "solve" a dependency graph with respect to versions + constraints, but note that this is relatively less common in the R universe.

[kenahoo]

Hi @kevinushey, your assertion

R and renv only support a single constraint on the package version.

is not true - see https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Package-Dependencies

A package or ‘R’ can appear more than once in the ‘Depends’ field, for example to give upper and lower bounds on acceptable versions.

[kevinushey]

Thanks, I wasn't aware of that. Although the wording implies that versions would have to be declared with multiple separate fields, e.g.

dplyr (>= 1.0.0), dplyr (< 1.1.0)

Either way, the truth remains that upper-level version constraints are uncommon.

It also doesn't mention whether that remains true for other fields (e.g. Imports, LinkingTo).

[kenahoo]

@kevinushey I would maintain that upper-bound constraints may be uncommon in packages uploaded to CRAN itself, but they are likely very common for privately-maintained packages and applications. They are very common in my organization, for example.

In practice they are indeed used also in Imports, LinkingTo, etc.

I have assumed that the main purpose of renv is to help people maintain these private environments, not public CRAN packages, so getting this supported in renv does seem important.

[kevinushey]

The primary goal for renv is to make it easy to save and restore the current state of your R library, under the assumption that R packages have been installed via the "commonly-used" set of tools for package installation (install.packages, remotes, devtools).

Solving the dependency graph given a set of version constraints is a separate task that renv does not currently try to solve. However, if the user is able to get their R library into a "good" state (e.g. by manually installing the requisite packages they need), then renv at least promises that it can later restore that "good" state.

renv was built primarily with CRAN-centric workflows in mind, hence the reason why this work has not yet been explored.

[kenahoo]

I'm not trying to be a thorn here, but I'm not sure what you mean by "CRAN-centric workflows". If I'm writing a script or app in some environment, and I'm pulling CRAN packages into that environment, isn't that "CRAN-centric"? That's exactly renv's niche, right?

Certainly renv isn't (right?) targeting the development of CRAN packages themselves, which must maintain broad compatibility with other packages, so would never use these tightly pinned dependencies renv presumes.

The advice about getting a workspace into some "good state", by hook or by crook, is good, I agree that it's likely what we need to do until/unless better tools become available for that work.

BTW, I don't think redirecting people to pak is a great idea - it doesn't seem like a mature or even very actively developed project at this point.

[kevinushey]

Sorry, by CRAN-centric I mean for projects that are at least in part trying to track what is "latest" on CRAN, while having a mechanism for making updates a little safer (that is, you have a private library that is in a "good" state that can be restored if you attempt to update packages and discover something is broken or incompatible with your project in some way).

Version constraints could also be a similarly useful vehicle, but I think in practice even conscientious package authors who try not to break things in patch releases occasionally do so, so in the end the version constraints don't protect you as much as you might've hoped. (Many package authors also don't subscribe to semantic versioning as well) That's definitely less true in internal environments where these sorts of concerns are better managed, though.

BTW, I don't think redirecting people to pak is a great idea - it doesn't seem like a mature or even very actively developed project at this point.

I would describe it as maturing -- I know a lot of folks on the r-lib team are testing and using it fairly extensively internally, but I wouldn't be surprised if there are still some outstanding issues. (Note that since pak is effectively a front-end to a whole ecosystem of R packages, you won't see as much activity directly on the pak repository.)

I'm hopeful that eventually pak and renv will be able to work together in a way where pak can be used for "general" package installation, while renv can manage other concerns (e.g. the lockfile; other relevant project-specific settings and options)

[kenahoo]

Ah, I see - so the expectation would be that users either use a completely static set of deps, or try to track CRAN periodically. For people doing the latter, I do think it would also be very helpful to allow dependency ranges in the same way packages can specify them. If pak is mature enough for this use case, it would seem natural to delegate to it in renv for this purpose.

Alternatively (or additionally?) it would be good to warn users who use version criteria that they're being ignored. I know that caught our team by surprise when we found out versions we thought were prohibited were silently being used anyway.

[tannerntannern]

it would be good to warn users who use version criteria that they're being ignored. I know that caught our team by surprise when we found out versions we thought were prohibited were silently being used anyway

Definitely. Wasted a day trying to understand this issue, which could have been avoided if it was clear on this page: https://rstudio.github.io/renv/articles/packages.html

[kenahoo]

I would describe it as maturing -- I know a lot of folks on the r-lib team are testing and using it fairly extensively internally, but I wouldn't be surprised if there are still some outstanding issues.

Unfortunately this doesn't match my experience - I just downloaded the latest version of pak and used it for my very most basic use case, and it fails with an uninformative error message:

> install.packages('pak')
Installing package into '/Users/kwilliams/R/library/4.0'
(as 'lib' is unspecified)
######################################################################## 100.0%

The downloaded binary packages are in
	/var/folders/zp/hj5hqfw970z0_78mrb_802lm0001z9/T//RtmpzfqB2e/downloaded_packages
> dir.create('llib')
> pak::local_install_deps(lib='llib')

`pak` will create its private package library in
`/Users/kwilliams/Library/Caches/R-pkg/lib/4.0`. 
It will try to copy packages from your regular library
See `?pak_setup()` for alternatives.

Do you want to continue (Y/n)? y

Creating private lib in `/Users/kwilliams/Library/Caches/R-pkg/lib/4.0`...
Copying package `assertthat`
Copying package `base64enc`
Copying package `callr`
Copying package `cli`
Copying package `cliapp`
Copying package `crayon`
Copying package `curl`
Copying package `desc`
Copying package `filelock`
Copying package `glue`
Copying package `jsonlite`
Copying package `lpSolve`
Copying package `pkgbuild`
Copying package `pkgcache`
Copying package `prettyunits`
Copying package `processx`
Copying package `ps`
Copying package `R6`
Copying package `rematch2`
Copying package `rprojroot`
Copying package `tibble`
Copying package `fansi`
Copying package `prettycode`
Copying package `progress`
Copying package `selectr`
Copying package `withr`
Copying package `xml2`
Copying package `digest`
Copying package `rappdirs`
Copying package `rlang`
Copying package `uuid`
Copying package `backports`
Copying package `ellipsis`
Copying package `lifecycle`
Copying package `magrittr`
Copying package `pillar`
Copying package `pkgconfig`
Copying package `vctrs`
Copying package `hms`
Copying package `stringr`
Copying package `utf8`
Copying package `stringi`

Created private lib in `/Users/kwilliams/Library/Caches/R-pkg/lib/4.0`...
Error: callr subprocess failed: Invalid parent_resolve callback

I would very much like pak to work, because it aims to solve real problems that my team has, but I feel like I gave up on it a couple of years ago when encountering similar issues.

[kenahoo]

Remembering another issue with pak that blocks us from using it: r-lib/pak#119

[kevinushey]

That is unfortunate. :-/

For what it's worth, the version of pak on CRAN is rather out of date now; the development version can be installed with:

install.packages("pak", repos = "https://r-lib.github.io/p/pak/dev/")

and should (hopefully) behave better. However, I'm not sure if the specific issue you mention in r-lib/pak#119 is resolved.

[kenahoo]

Thanks Kevin - I installed the latest from GitHub and it looks like it got past the parent_resolve issue, but still has the r-lib/pak#119 issues so we can’t hit our local repository.

I appreciate you taking the time to help!