Best Practice for Pipelines On Top of Packages

[kyle-messier]

Prework

• I understand and agree to help guide.
• I understand and agree to contributing guide.
• New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

I am curious what the recommended best-practice is for utilizing targets with a workflow/pipeline style package. I have a project in development that (1) downloads, (2) processes, (3) models, (4) post-process, (5) checks for new data,(6) rinse and repeat. So, targets is a natural fit for the pipeline, but I am not sure where it fits in the package. My ideas are:

1. Target Markdown workflow and add the full pipeline as a vignette
2. Create a separate target branch that is more or less a co-equal branch to main
3. Separate repo all together

Thanks!

[wlandau]

I have found it easiest to have at least some separation. Maybe the package implements the kinds of functions that each target calls, and maybe the package even writes a _targets.R file. But then the actual instance of _targets.R and any scripts sourced from _targets.R are best to put in a separate project outside the source code of your package. https://books.ropensci.org/targets/packages.html might help too.

[wlandau]

FYI @Spatiotemporal-Exposures-and-Toxicology I converted this to a discussion.

[kyle-messier]

The separate repos has made it easy enough to develop. For pipelines that are not explicitly dependent on 1 package, I could see this being the way. However, it would be nice to have it is a vignette. What do you think if we made the pipeline a target markdown and then put _targets.R and supplementary functions specifically for targets and outside the scope of the package in the .Rbuildignore, such that any R package building doesn't have to deal with it?

[wlandau]

I know Target Markdown exists, but it's a bit awkward to flatten out a targets pipeline into a single file. Might be better if that vignette walks through the functions of the workflow from end to end in a simple case. You could explain how the flow of the project goes in a scaled-down example without needing the actual pipeline to be there.

[kyle-messier]

Sounds good, thank you!