Open
Description
Describe the feature you'd like
A feature to selectively execute a sagemaker pipeline step which has upstream dependencies when all the dependent parameter for the step is passed via a dict.
Presently using sagemaker.workflow.selective_execution_config.SelectiveExecutionConfig to execute a pipeline with dependencies gives a client error.
ClientError: An error occurred (ValidationException) when calling the StartPipelineExecution operation: Invalid
SourcePipelineExecutionArn: null. The selected steps [<step name>] have dependencies on [<step name>, ...]The expected behavior would be to execute the step successfully when the dependent parameters are passed.
How would this feature be used? Please describe.
- Selective execution of steps which are downstream which have many dependencies. Executing all the dependencies seems wasteful.
Describe alternatives you've considered
- Execute all dependencies, current feature.
- Create a temporary new pipeline with the specification for the step by removing the dependency and executing it.
Additional context
sagemaker sdk version: 2.208.0
Any alternative or suggestion would be helpful.