Improve support for pipeline chaining #6574
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds several features to facilitate pipeline chaining based on workflow inputs/outputs
Demo project: bentsherman/pipeline-chaining-demo
The JSON output is very similar to the lineage record (
lid://<workflow-run-hash>#output), so maybe we could use that instead. The lineage record has the advantage of using LIDs for files, but the JSON structure doesn't quite match the params file.With a bit more magic we could even make the pipeline chain as simple as a Bash pipeline:
But the main thing for now is the basic reading/writing of inputs/outputs, then enhanced validation with schemas and record types.