Replies: 7 comments 28 replies
-
|
I did not realize that the versioning system in For context, the primary reason I made the switch was to eliminate the egregious performance penalties from calling the AWS API in large pipelines. In I solved this problem by caching the results of a paginated LIST request, which reduced AWS HTTPS calls by 1000x and time spent by >60x. But unfortunately, ListObjectsV2 is not version-aware. The only alternative that might work is ListObjectVersions, which would download every single version of every single object in the pipeline, which would be an unmanageably large payload. So as a side effect, |
Beta Was this translation helpful? Give feedback.
-
Agreed, metadata is king. But now in the case for |
Beta Was this translation helpful? Give feedback.
-
|
Hmm, this is definitely going to require us to consider different strategies. I wonder if it is possible for us to implement a version where targets are named by hash in the S3 store, rather than by target name, and drop the versioning feature in the S3 bucket entirely. (This would have the added benefit of allowing the store to move between S3 providers, which is not possible for S3 object version history). |
Beta Was this translation helpful? Give feedback.
-
This is how Rich Fitzjohn's Even if it were feasible, it would create a lot of "garbage" objects: old files whose names are hashes and which no longer belong to any current target. This would have the nice effect of supporting the kind of seamless collaborative versioning you are looking for, but it would also cause storage usage to balloon in many cases, both in gigabytes used and the number of files. The large number of files would put a strain on local file quotas and slow down the AWS S3 paginated LIST request that currently helps I describe all this not to argue, but because I think someone else might ask the same question, and I want to be able to read this comment and remember the specific reasons why I made these design choices. I wonder if Docker-like layering would help your team. Docker uses a union file system to implement its layering capabilities (e.g. Union FS or Aufs, I think the choice is pluggable.) Not sure if I'm off base. |
Beta Was this translation helpful? Give feedback.
-
|
I'm not going to argue for you to change directions, but I must say this change surprised me because of how well suited the S3 versioning feature was for this workflow. I really thought it was one of the primary design goals for it. I don't know how widespread the user base is of this feature but I would be curious what their use cases are and if this will be an issue for others. |
Beta Was this translation helpful? Give feedback.
-
If it seems like I am belaboring the point, it's because I think someone else is going to ask this same question several months from now, and I want to be able to look back on this discussion and remember why I did what I did.
I was going for a "time machine" effect so you could revert the metadata and read the matching output data, but I was not thinking about the possibilities for multiple people collaborating on a single data store.
At my workplace, our sys admin created a bucket for all of us, and we are only allowed to access data at a prefix specific to each user. So we would not be able to exploit the old |
Beta Was this translation helpful? Give feedback.
-
|
The thing that would make it possible to roll back to the previous behavior is some kind of inexpensive way to check the ETags of multiple S3 object versions. So if I have a list of 3000 AWS S3 objects and a version ID for each object (whether current or old), I would need a way to check the ETag of each of those specific 3000 object versions, while only using 3 API calls and without running through any of the objects or versions outside the list. So far I don't think AWS can do this, except possibly in an S3 batch operation, which looks like it is high-overhead and requires an esoteric set of permissions. And I would also want consistent behavior with Google Cloud so |
Beta Was this translation helpful? Give feedback.
-
|
One possibility is to allow for the option of using Our other current idea is to stop targets S3 versioning and and create our own solution of having our own proxy object types that refer to saved files on S3. This would have the benefit of allowing some other caching options, as well, but it would only make sense for specific high-cost targets. In this case we would lose general versioning capabilities for the rest of the pipeline that would allow easy time-machine-like comparison across versions with https://github.com/ecohealthalliance/relic/ |
Beta Was this translation helpful? Give feedback.
-
|
As you alluded to, |
Beta Was this translation helpful? Give feedback.
-
|
Is the problem with the idea of using ListObjectVersions with version-id-marker that you have no proper way to limit which objects you get back (like you only want a subset not all objects), or is the problem that you get more versions than you want? I would assume the bucket is only used for targets, so all objects should be valid to get? And with regards to unwanted versions - is not the point of versioning them that you want to keep them around? I can see that the API is not ideal though :) I found this blog-post https://tech.trivago.com/post/2018-09-03-s3-versioning where they were grappling with enormous buckets and this API call. Did any of you look at the possibility of using the version-id-marker as a sort of caching key. So for example, if one downloads all versions for all objects using ListObjectVersions and paging (until isTruncated is false), and then later comes back after something has changed, and uses the version-id-marker from last time, do you only get the new updates since last? |
Beta Was this translation helpful? Give feedback.
-
|
I was about to ask if, in the long-term, you would be open to custom repository back-ends other than "aws" and "gcp" that might allow for custom behaviors. However, it occurs to me that this might be entirely possible to implement using custom formats, where read and write functions access a local or cloud key-value store and just save the keys to disk. I'll toy with this a bit, but I'm sure it is also an "off-label" use case so I'll check before trying anything serious with it. |
Beta Was this translation helpful? Give feedback.
-
|
[Continues thinking out loud] On the other hand, a format-only approach to this would mean that targets would be unaware of the storage/caching mechanism, so all the status and monitoring info would be wrong, and targets would error if the cache were cleared without clearing the key files. I can think of some solutions (e.g. a set-up/sync function in the targets script), though they may be pretty hacky. To be continued...🤔 |
Beta Was this translation helpful? Give feedback.
-
|
What did you end up doing to resolve it at your end, downgrade to before 1.4.0? |
Beta Was this translation helpful? Give feedback.
-
|
We're in-progress with this. Some projects are sticking with pre-1.4.0 versions (we run everything with fixed versions with |
Beta Was this translation helpful? Give feedback.
-
|
We're also using S3 versioning with a shared bucket for multiple users on a project, so this is an interesting discussion to me. Our server is running locally (minio) and we're also doing As I can understand the change was done to speed up working with AWS. But the side-effect is that now targets determines if a pipeline is up-to-date differently? Before it was considering local metadata vs AWS store, and now it ignores local metadata? |
Beta Was this translation helpful? Give feedback.
-
|
Reading the description again it says "For a cloud target to be truly up to date, the hash in the metadata now needs to match the current object in the bucket". Does that mean that user A could be running |
Beta Was this translation helpful? Give feedback.
-
|
Yes, that's the case. |
Beta Was this translation helpful? Give feedback.
-
|
Ouch😬. Thanks for verifying. I guess we'll have to downgrade. |
Beta Was this translation helpful? Give feedback.
-
Looking back on this now. The approach you describe is apparently called "content-addressable storage" (CAS). Like I mentioned, It would be better to be able to opt into CAS regardless of the other storage settings (e.g. AWS vs local). I am still not sure this is feasible for |
Beta Was this translation helpful? Give feedback.
-
|
We are very much interested in a CAS approach for targets with either a locally or remote/cloud solution. DVC is already built this way https://dvc.org/doc/user-guide/project-structure/internal-files#structure-of-the-cache-directory (works both GC / AWS and local shared directory as the remote). I am following this thread with interest 👍 |
Beta Was this translation helpful? Give feedback.
-
Actually, a better way to do this is with if (tar_active()) {
# only runs the on tar_make(), tar_make_clustermq(), or tar_make_future():
synchronize_pointer_files()
}
list(targets::tar_target(...), ...) |
Beta Was this translation helpful? Give feedback.
-
|
Hi everone, I'm also very much interested in support for a CAS-like cache that could be shared between myself and my team mates. After reading this thread, I experimented a bit with the idea of replacing the files in Minimal proof-of-conceptI started out with a very minimal library(targets)
list(
tar_target(foo, 1),
tar_target(bar, foo + 1)
)I tried to jam all of my code into a This is my proof of concept code ( library(dplyr)
# create a _targets/cas dir where the object files will reside
dir.create(file.path(targets::tar_path_store(), "cas"), showWarnings = FALSE, recursive = TRUE)
# check if a path is a symplink
is_symlink <- function(paths) isTRUE(nzchar(Sys.readlink(paths), keepNA=TRUE))
# replace a single file in _targets/objects with a symlink to the previously
# stored data in _targets/cas
force_link_to_cas <- function(hash, name) {
cas_dir <- file.path("..", "cas")
cas_source_file <- file.path(cas_dir, hash)
object_target_file <- do.call(targets::tar_path_target, list(name))
if(file.exists(object_target_file)) {
file.remove(object_target_file)
}
file.symlink(cas_source_file, object_target_file)
paste0(object_target_file, ">", cas_source_file)
}
# for any stems in the meta file where there is a matching file (correct hash)
# in the cas, replace the file in _targets/objects/ with a symlink to the
# file in _targets/cas
targets::tar_meta() %>%
filter(type == "stem") %>%
select(name, data) %>%
rowwise() %>%
mutate(link = force_link_to_cas(data, name))
# run the pipeline as usual
targets::tar_make()
# move any newly created files in _targets/objects/ to _target/cas
# and replace by a symlink to this new cas entry
add_file_to_cas <- function(hash, name) {
cas_dir <- file.path(targets::tar_path_store(), "cas")
cas_target_file <- file.path(cas_dir, hash)
cas_dir_rel <- file.path("..", "cas")
cas_target_file_rel <- file.path(cas_dir_rel, hash)
object_source_file <- do.call(targets::tar_path_target, list(name))
if(!file.exists(cas_target_file) && !is_symlink(object_source_file)) {
file.rename(object_source_file, cas_target_file)
file.symlink(cas_target_file_rel, object_source_file)
}
paste0(object_source_file, ">", cas_target_file)
}
# for any stems in the meta file where there is a true file in _targets/objects
# move that file into the _targets/cas (and rename to its hash), and add a
# symlink from _targets/objects to this new entry in _targets/cas
targets::tar_meta() %>%
filter(type == "stem") %>%
select(name, data) %>%
rowwise() %>%
mutate(link = add_file_to_cas(data, name)) This proof of concept enables the following workflow:
Outstanding issues and questions
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for reminding me of this discussion and for offering a prototype.
This is the purpose of the Here is a prototype based on The only thing missing is a way to synchronize local pointer files. library(targets)
cas_format <- tar_format(
read = function(path) {
# Step 1: look up the hash from the local pointer file.
hash <- readLines(path)
# Step 2: use the hash to get the data from CAS storage.
data <- readRDS(file.path("cas", hash))
},
write = function(object, path) {
# Step 1: save the object to a local temporary file.
temp <- tempfile()
on.exit(unlink(temp)) # Clean up the temporary file on exit or error.
saveRDS(object, temp)
# Step 2: compute the hash on the temporary file.
hash <- secretbase::siphash13(file = temp)
# Step 3: move the file to the CAS system if needed.
new_path <- file.path("cas", hash)
if (!file.exists(new_path)) {
fs::dir_create(dirname(new_path), recurse = TRUE)
fs::file_move(path = temp, new_path = new_path)
}
# Step 4. Save the hash to the local pointer file.
writeLines(hash, path)
}
)
tar_option_set(format = cas_format)
list(
tar_target(x, seq_len(2)),
tar_target(y, x, pattern = map(x))
) |
Beta Was this translation helpful? Give feedback.
-
|
Syncing pointer files is tricky because the library(targets)
cas_format <- tar_format(
read = function(path) {
# Step 1: look up the hash from the local pointer file.
hash <- readLines(path)
# Step 2: use the hash to get the data from CAS data storage.
readRDS(file.path("cas/data", hash))
},
write = function(object, path) {
# Step 1: save and hash the actual data to a local temporary file.
temp_data <- tempfile()
on.exit(unlink(temp_data)) # Clean up the temporary file on exit or error.
saveRDS(object, temp_data)
hash_data <- targets:::hash_file(temp_data) # hash_file() would need to be exported.
# Step 2: save and hash the pointer file to another local temporary file.
temp_pointer <- tempfile()
on.exit(unlink(temp_pointer), add = TRUE)
writeLines(hash_data, temp_pointer)
hash_pointer <- targets:::hash_file(temp_pointer)
# Step 3: move data to CAS system.
cas_data <- file.path("cas/data", hash_data)
if (!file.exists(cas_data)) {
fs::dir_create(dirname(cas_data), recurse = TRUE)
fs::file_move(temp_data, cas_data)
}
# Step 4: copy pointer file to CAS system.
cas_pointer <- file.path("cas/pointers", hash_pointer)
if (!file.exists(cas_pointer)) {
fs::dir_create(dirname(cas_pointer), recurse = TRUE)
fs::file_copy(temp_pointer, cas_pointer)
}
# Step 5: store pointer file in _targets/objects/
fs::file_move(temp_pointer, path)
}
)
tar_option_set(format = cas_format)
if (tar_active()) {
# Sync pointer files
unlink("_targets/objects", recursive = TRUE)
dir.create("_targets/objects")
meta <- tar_meta(fields = any_of(c("name", "data")))
sync <- meta[meta$data %in% list.files(file.path("cas/pointers")), ]
purrr::pwalk(
sync,
\(name, data)
fs::file_copy(
file.path("cas/pointers", data),
file.path("_targets/objects", name)
)
)
}
list(
tar_target(x, seq_len(2)),
tar_target(y, x, pattern = map(x))
) |
Beta Was this translation helpful? Give feedback.
-
|
An update: as of #1322, content-addressable storage (CAS) is now part of I think the only tricky problems left to the user are (1) garbage collection / retention policy, and (2) an efficient targets/R/tar_repository_cas.R Lines 1 to 238 in 4fe9da7 |
Beta Was this translation helpful? Give feedback.
-
|
FYI |
Beta Was this translation helpful? Give feedback.
-
Help
Description
My team recently realized that this change in targets 1.4 may have significant consequences for our cloud-based workflows:
We use S3 versioning with a shared bucket for multiple users on a project. It has been excellent for letting us share compute-intense targets and avoiding downloading large targets that can be skipped, even as each user works on different branches of a project where some targets may diverge. As long a version of the target in the bucket exists that matches the local metafile, it could be skipped, and overwriting the latest version wouldn't interfere with others' state. This also means CI builds of targets can take advantage of already-built targets without interfering with development. Under targets 1.3, this requires setting `repository_meta = "local", which is fine. However, does this mean that in 1.4+, this approach to collaboration will no longer work? Is there a way to recover the old behavior? Being able to collaborate this way has been the primary benefit of cloud-based versioning for us.
@emmamendelsohn
Beta Was this translation helpful? Give feedback.
All reactions