Support developer tests

[jeroen]

A cleaner pattern to distinguish between public tests and developer tests.

Developer tests only run in CI or devtools, and can be used for tests that are not appropriate for public infrastructure such as CRAN or other downstream use, e.g. tests that require special tooling, are flaky, or consume much resources.

[DavisVaughan]

Possibly there is room for an explicit opt in/out env var that is very specific to this feature, like:

run_dev_tests <- function() {
  if (env_var_is_true("RUN_DEV_TESTS")) {
     return(TRUE)
  }

  if (env_var_is_false("RUN_DEV_TESTS")) {
     return(FALSE)
  }

  env_var_is_true("CI") || env_var_is_true("NOT_CRAN")
}

[jeroen]

Thinking a bit more about this after todays dicussion about how this works for reverse dependency checks. Perhaps the dev tests should be opt-in per package. So the variable should be named e.g. JSONLITE_RUN_DEV_TESTS to opt-in to dev tests from jsonlite only.

For example, if I want to run a reverse dependency check for curl, I also may want to run the developer tests for httr and httr2 but not for all the random packages with api wrappers that require custom auth tokens and so on.

[JosiahParry]

Just chiming in to say that I would really like to see this capability added to testthat. My particular use case is that my package arcpbf is used by arcgislayers and I want arcpbf to have integration tests for GHA CI but not to actually be tested on CRAN.

[hadley]

A few other questions to consider:

• Would we suggest that if your R code is in R/foo.R your developer tests live in tests/testthat/dev-tests-foo.R?
• How does this impact devtools:use_r() and devtools::use_test()? Would we need a new use_dev_test()?
• Does context_name() also need adjustment?
• What happens if you use a snapshot test from one of these files? Does it work at all currently?
• How does snapshot cleanup work? i.e. if you don't run the developer tests do we need to make changes to the snapshot cleanup logic so that they don't get deleted?
• Given the snapshot issues, would it be simpler to put them in a new test directory (e.g. tests/testdev/test-foo.R)?
• If you want to add per package env vars, I'd want to see a thorough write up. Our current revdep check system is certainly far from perfect, but we do fully understand the limitations.

I think I'm overall sceptical that this is the correct approach compared to adding the appropriate skip calls to each individual test. I realise that is a lot of work, but it's really hard to reason about snapshots without it. (And it reduces the amount of code you need to read to tell if an individual test will run or not.)

[jeroen]

I think I'm overall sceptical that this is the correct approach compared to adding the appropriate skip calls to each individual test. I realise that is a lot of work, but it's really hard to reason about snapshots without it. (And it reduces the amount of code you need to read to tell if an individual test will run or not.)

Okay perhaps this idea then is not a good fit for the testthat ecosystem incl devtools, revdepcheck, and snapshots.

What I find it appealing is no so much the amount of work, but that to me it makes more sense to reason about developer test and deployment tests, rather than special casing each test with skip-on-xyz. But it see that it complicates the notion of snapshots in particular.

But perhaps it makes more sense to do this on the package level by passing a filter in test_check() to opt-in to a subset of the test.

[hadley]

I'm willing to be persuaded, but I think there's quite a lot of work more to do, and I suspect there are easier ways to get what you want.