-
Notifications
You must be signed in to change notification settings - Fork 621
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wait until pipeline completes before publishing #5239
Comments
The problem is that deferring all publishing to the end of the workflow can increase the total runtime significantly. The option you propose would punt the trade-off to the user, but I wonder if we can do better. We have discussed before, simply deleting the output directory in the Another option I'm considering is to treat publish tasks like regular tasks and allow them to have an error strategy. There is already some basic retry logic, and the |
I think this is OK, it's a choice the user makes to defer publishing to the end at the cost of runtime. If it's something they want that's fine.
I'm not a fan of this option because it doesn't work if the pipeline doesn't end gracefully or if anything prevents files being cleared up. It just feels...inefficient? Creating files just to delete them later? The main downside is this directly conflicts with any clearup methods that get implemented, although it wouldn't be hard to make the options mutually exclusive.
This seems like a good idea, also it keeps syntax consistent which is always a bonus. Of course, this is low priority since with the new output DSL we can use channel operators to achieve this 🥳 |
New feature
Currently, Nextflow will publish files as it proceeds through the pipeline. If it fails, it will halt leaving some files in place. This creates messy output structures that need to be cleaned after running the pipeline.
Usage scenario
It may be desirable to wait until we know the pipeline has completed successfully before publishing to make sure all files will be created successfully and isolate pipeline running from output file creation.
This is sort of possible with the new output syntax because you could use channel operations to 'hold' the files until all are complete, e.g. this will use
groupTuple
to hold the files until they are all emitted, essentially blocking publication until the pipeline has completed.However we may just wish to force it to happen after the pipeline completes but before the
workflow.onComplete
happens.Suggest implementation
This could be an additional option to the output section:
Or some other way of holding the files back until the last moment.
The text was updated successfully, but these errors were encountered: