Support Multipart CWL Deployment

[fmigneault]

Description

When a CWL consists of a class: Workflow composed of multiple CWL files, the deployment process can be painful, since users must deploy each child step (to ensure they resolve without missing dependencies) before deploying the final root workflow.

Furthermore, although CWL can be deployed on their own, embedding it within the executionUnit allows the deployment to provide additional/complementary metadata and process description details. However, embedding the CWL within the process description must be done manually by the user, or using the CLI that happens to offer this capability.

Instead, Content-Type: multipart/* could be used. This would allow uploading all workflows steps simultaneously, as separate CWL parts, as well as any additional process description metadata. Using multipart/related and/or multipart/mixed (either in conjunction or separately), it is also possible to employ Content-ID (https://datatracker.ietf.org/doc/html/rfc2392.html) to provide relationships between the various child-CWL and their respective descriptions. It is also possible to use a nested hierarchy, as follows:

• multipart/mixed
  • root CWL workflow application/cwl+json
  • multipart/related
    • step1 CWL CommandLineTool application/cwl+json
    • step1 ProcessDescription application/json
  • step2 CWL CommandLineTool application/cwl+json

To Do

• Support the multipart/* process deployment
  • Support for a plain CommandLineTool along its ProcessDescription metadata
  • Support for a workflow, which must deploy each step first, and then the root workflow
• Supported nested multipart/*
• Consider Content-ID to match corresponding CWL and ProcessDescription definitions (since they are not necessarily next to each other in the contents)
• Consider {cid:[...]} cross-references if needed
• Consider if multipart/related must be enforced as top level ? https://datatracker.ietf.org/doc/html/rfc2387
  • The definition Content-Type: multipart/related; type="..."; start="<cid> is available in the top-level of the request, which can indicate which is the root workflow.
• Consider Content-Disposition of each part for relevant CWL file references used for run: of the workflow
• Update the CLI to allow a list of CWL, which would generate the relevant multipart contents to deploy them

References

• n/a