README.Rmd

---
output: github_document
---

<!-- README.md is generated from README.Rmd. Please edit that file -->

```{r}
#| include: false
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
```

# nestedtune <a href="https://nestedtune.tidymodels.org"><img src="man/figures/logo.png" align="right" height="138" alt="nestedtune website" /></a>

<!-- badges: start -->
[![R-CMD-check](https://github.com/tidymodels/nestedtune/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/tidymodels/nestedtune/actions/workflows/R-CMD-check.yaml)
[![Codecov test coverage](https://codecov.io/gh/tidymodels/nestedtune/graph/badge.svg)](https://app.codecov.io/gh/tidymodels/nestedtune)
<!-- badges: end -->

The best score from ordinary tuning is optimistic, because the same
resamples pick the winner. nestedtune runs nested cross-validation instead.
Each outer fold tunes a workflow on its own inner resamples with tune or
finetune, then fits the chosen settings and scores them on rows the tuning
never saw. The mean of the outer scores estimates how well the whole
procedure (tune, select, fit) performs on new data. Report it for the model
that `nested_final_fit()` builds.

## Installation

``` r
# install.packages("pak")
pak::pak("tidymodels/nestedtune")
```

## Example

```{r example, message = FALSE}
library(tidymodels)
library(nestedtune)

set.seed(1)
folds <- nested_resamples(
  mtcars,
  outside = vfold_cv(v = 5),
  inside = vfold_cv(v = 5)
)

wf <- workflow(
  mpg ~ .,
  rand_forest(mtry = tune(), min_n = tune()) |>
    set_engine("ranger") |>
    set_mode("regression")
)
grid <- expand.grid(mtry = c(2L, 5L, 8L), min_n = c(2L, 10L))

set.seed(2)
res <- nested_tune_grid(wf, folds, grid = grid)

# The number to report for the model you deploy.
collect_metrics(res)

# The model to deploy, the same procedure run once more on all the data.
set.seed(3)
final <- nested_final_fit(wf, res)
predict(final, new_data = mtcars[1:3, ])
```

## Supported resampling designs

The table covers 15 resampling functions from rsample. It says whether each
one works as the outer loop and as the inner loop of a nested design.

| Function | Outer loop | Inner loop |
|---|---|---|
| `vfold_cv()` | Yes | Yes |
| `mc_cv()` | Yes | Yes |
| `group_vfold_cv()` | Yes | Yes |
| `group_mc_cv()` | Yes | Yes |
| `clustering_cv()` | Yes | Yes |
| `bootstraps()` | Refused | Yes |
| `group_bootstraps()` | Refused | Yes |
| `loo_cv()` | Refused | Refused |
| `apparent()` | Refused | Refused |
| `validation_set()` | Yes[^validation] | No |
| `permutations()` | Refused | Refused |
| `rolling_origin()` | Yes | Yes |
| `sliding_window()` | Yes | Yes |
| `sliding_index()` | Yes | Yes |
| `sliding_period()` | Yes | Yes |

Yes means that the test suite runs the design in that loop through
`nested_tune_grid()`, with the arguments those tests use, and that every
outer fold completes. The design in the other loop is a v-fold design, or a
time-series design for the four time-series rows.

Refused means that nestedtune stops with an error before any fold runs,
whether `nested_resamples()` or `rsample::nested_cv()` built the design. No
means that the design cannot be built in that loop. The "Differences from
rsample" section of `?nested_resamples` gives the reason for each Refused
and No cell, and the rules for a single split in either loop.

[^validation]: Build the validation set before the call, with
`split <- initial_validation_split(data)` and `validation_set(split)`. Then
pass `rbind(training(split), validation(split))` as `data`, because those are
the rows the validation set refers to. A call such as
`outside = validation_set()` cannot be built, because the function takes a
split rather than a data frame.

## Supported tuning functions

The table lists tuning functions from tune, finetune, workflowsets and
tidyclust, and the nestedtune function that does each one's job on a nested
design.

| Function | nestedtune function | Supported |
|---|---|---|
| `tune::tune_grid()` | `nested_tune_grid()` | Yes |
| `tune::tune_bayes()` | `nested_tune_bayes()` | Yes |
| `tune::fit_resamples()` | `nested_fit_resamples()` | Yes |
| `finetune::tune_race_anova()` | `nested_tune_race_anova()` | Yes |
| `finetune::tune_race_win_loss()` | `nested_tune_race_win_loss()` | Yes |
| `finetune::tune_sim_anneal()` | `nested_tune_sim_anneal()` | Yes |
| `workflowsets::workflow_map()` | `nested_workflow_map()` | Yes |
| `tidyclust::tune_cluster()` | | No |

Yes means that nestedtune exports a nested counterpart that does the row's
job on a nested design. The test suite runs it with `vfold_cv()` in both
loops, and every outer fold completes. `nested_fit_resamples()` and
`nested_workflow_map()` run no search of their own. The first scores a
workflow with nothing to tune, and the second runs each workflow of a set
through one of the other six functions. No means that no nestedtune function
does the row's job. For `tune_cluster()`, `tune::last_fit()` does not accept
a clustering workflow, so no nestedtune function can score the outer fold.
Selecting on a sum-of-squares metric also tends to pick the largest number of
clusters in the grid. When it does, the estimate describes the grid and not
the data.

The resampling table's Yes cells were tested through `nested_tune_grid()`.
The "Time-series designs" section of `?nested_resamples` says which
time-series designs the other six functions are tested on. [Choosing the
inner tuner](https://nestedtune.tidymodels.org/articles/tuners.html) runs
the Bayesian search, the two racing searches and simulated annealing on one
example.

Learn more:

- [Nested cross-validation](https://nestedtune.tidymodels.org/articles/nested-cv.html),
  the path from a design to a write-up.
- [What the estimate means](https://nestedtune.tidymodels.org/articles/estimate.html),
  which quantity the nested number is and how far to trust it.
- [Choosing the inner tuner](https://nestedtune.tidymodels.org/articles/tuners.html),
  the Bayesian search, the two racing searches, simulated annealing and a
  set of workflows on one design.
- [Reading the results](https://nestedtune.tidymodels.org/articles/results.html),
  one run read from start to finish: the estimate, the selections, the
  predictions and the plots.
- [Running the outer loop in parallel](https://nestedtune.tidymodels.org/articles/parallel.html),
  the same call on a pool of mirai daemons.