Feature Request: Run jobs through the future API

[zeehio]

Idea:

• With brickster&DBI I can use dplyr to write SQL queries. That's great!
• With a brickster&future I can use purrr/furrr to parallellize loops on databricks. Let's do this!

The future API is a well designed API to distribute R work among workers. It's nice because it allows using furrr (equivalent to purrr) but with futures, it supports nice progress bars, and it is performant and robust. It's been around for years and it is very well maintained.

https://future.futureverse.org/

Future separates "what to parallelize" which is defined by a package developer (e.g. "As a pkg developer I want this expensive computation to run in parallel") from "how to parallelize" (e.g. As a package user I want "that parallelization to use 4 cores from my laptop" or "to run this heavy thing in a Slurm cluster of computers").

It provides documentation on how to define a new "future backend"

https://cran.r-project.org/web/packages/future/vignettes/future-6-future-api-backend-specification.html

I'd love to have a future backend that sends heavy calculations to a databricks cluster.

[zacdav-db]

Hey,

I think the preference would be to look at mirai and see if thats possible, which has been on my (very long) todo list.

The downside is there is a huge amount of complexity and potential dependencies.
There's an element of running things on Databricks thats currently covered by the REPL too.

If you have an example of how the pseudo-code would look at behave I think thats the best place to start, and we can work backwards towards what idea makes the most sense.

[zeehio]

Oh mirai is nice as well. I have seen that the future package supports a mirai future backend, so mirai would work well for future users as well.

I took a quick look to mirai. From what I have seen (and I may be wrong) Mirai transfers the expressions and environment to its daemons/clusters via a TCP connection (with TLS if encryption is required). I don't think it is possible to expose a TCP connnection from databricks. That looks like a challenge to me, although maybe you already know how to address it.

I'll do my best to provide some pseudo-code examples :-)

[zeehio]

Run some code on a databricks cluster and retrieve the results

This is useful to just dispatch some code to a compute cluster and get the results back.

library(future)
plan(databricks_cluster(cluster_id="xxx"))

# We must assume the function is self-contained, 
# and the required libraries are available in the cluster
slow_or_heavy_function <- function(x) {
    Sys.sleep(10)
    x + 1
}

# Create a future:
f <- future(slow_or_heavy_function(1:10))
# The slow_or_heavy_function runs on the cluster
# Get the value
y <- value(f)

Parallellize code on a databricks compute cluster

If we can dispatch one execution we can also parallellize many function calls. If the compute cluster has multiple cores it would be great to use them all:

library(future)
library(furrr)
plan(databricks_cluster(cluster_id="xxx"))

slow_or_heavy_function <- function(x) {
    Sys.sleep(10)
    x + 1
}

# This creates 10 futures, gets their values and stores them in y.
# I would expect those 10 futures to be evaluated in parallel in the cluster.
y <- furrr::future_map(1:10, slow_or_heavy_function)

Running code on an IDE connected to a remote databricks cluster

An interesting alternative approach would be to be able to use either Positron, RStudio Server or some IDE and run an interactive R session at a databricks session cluster. The db_repl() function is nice, but it has some limitations:

• The IDE variable explorer / environment tab does not show the variables in the notebook
• The "up arrow" shows me history of previous commands before the repl was started, not the previous commands I used in the R session
• Lack of good autocompletion (as far as I know)

This last part seems much harder, but the first two would make it easier to work with databricks from RStudio