[general] Garbage collection

[rsangole]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Will,

A garbage_collection argument is available in two places - the crew controller, and the tar_option_set. How should I think about this argument? Could you help me understand the difference between the two arguments?

targets::tar_option_set(
        packages = c(
            ...
        ),
        controller = crew::crew_controller_local(
            workers = 2,
            garbage_collection = TRUE, # <<---
        ),
        garbage_collection = TRUE # <<---
    )

[wlandau]

In current CRAN targets, tar_option_set(garbage_collection = ...) sets the default value of garbage_collection for any tar_target() calls made beyond that point. In tar_target(), garbage_collection = TRUE simply says to run garbage collection when it comes time to run that target.

I realized gc() was a huge bottleneck, so in development targets, these arguments now do different things. garbage_collection in tar_target() means the same thing, and for back compatibility, its default value is isTRUE(tar_option_get("garbage_collection")). In tar_option_set(), garbage_collection is now a positive integer to specify how often to run garbage collection independently of anything set in tar_target(). For example, tar_option_set(garbage_collection = 10) says to run gc() every tenth target, both on the local process and each parallel worker.