Ignored IAM role passed to Pipeline Steps breaks Local Mode

[lorenzwalthert]

Describe the feature you'd like

In non-local mode, I can pass a role to each step of my SageMaker Pipeline and this role will be assumed in that step. Hence, I can use different roles for different steps. This helps scoping permissions narrowly (for security) as well as working around the IAM limit of 10 managed policies per role and 10240 bytes character limit for inline policies, which are a problem once the role that is assumed in the pipeline needs too many permissions.

    # processor step
    processor_select_delivery_periods = sagemaker.processing.Processor(
        entrypoint=["bash", "processing.sh"],
        role= role_arn_for_processing_job, # <- this is ignored in local mode
        instance_type="ml.m5.large",
        instance_count=1,
        sagemaker_session=signals.sagemaker.pipeline_session.cache,
    )

    step_processing_args = processor_select_delivery_periods.run(
        arguments=...,
    )

    step_processing = sagemaker.workflow.steps.ProcessingStep(step_args=step_args,  ...)

    # training step 
    training = sagemaker.estimator.Estimator(
    	image_uri=image_uri,
    	role=role_arn_for_training_job, # <- this is ignored in local mode
    	instance_type ='ml.5.large',
    	instance_count=1,
    	sagemaker_session=signals.sagemaker.pipeline_session.cache,
    )
    
    step_training_args = training.fit(
        arguments=...,
    )

    step_train_model = sagemaker.workflow.steps.TrainingStep(step_args=train_args, ...)

   # finally build the pipeline
    pipeline = sagemaker.workflow.pipeline.Pipeline(
        steps=[
            step_processing,
            step_train_model,
        ],
        sagemaker_session=... # pass a local session if run locally, otherwise None

    )

However, the argument role seems not respected in local mode. Even worse, not even the IAM role defined at pipeline level seems to be respected

        upsert_response = pipeline.upsert(
            role_arn='arn:/xxx/yzzc',
            description=description or "",
            tags=tags,
        )

The credentials seem to be purely generated from the sagemaker_session passed to Pipeline:

sagemaker.workflow.pipeline.Pipeline(
        name=str(name),
        parameters=list(parameters),
        steps=[step_a, step_b],
        sagemaker_session=sagemaker_session
)

From looking at the source code of this SDK, in my understanding, the local mode's docker-compose file (one per step) contains credentials via environment variables (AWS_*) that are then used in the container. Before generating these credentials, one would need use sagemaker_session to assume the role passed to a step and then generate the credentials from these roles instead of directly from that session.

How would this feature be used? Please describe.

The inconsistency between local and remote execution breaks the highly efficient local developer workflow:

• Use per step roles makes sense from security and IAM limitations for large policies (both managed and unmanaged).
• However, it's practically unusable with local mode, a key ingredient of efficient pipeline development, as the roles are not respected in local execution and a local developer hence needs a role with the union of all permissions from all rules in remote execution, which defeats the purpose of per step roles.

Describe alternatives you've considered

• Refrain from using step-specific roles, i.e. one role for all steps, which means I'll face IAM restrictions.
• Splitting my pipeline into multiple pipelines since it contains many steps that are only loosely related. Drawback: Need another orchestration tool on top, seems an overkill.
• Not using local mode anymore: Not really an option, since without local mode, I get very slow in developing pipelines.

Additional context

I opened an AWS Support case: 174841854500356.

[rohangujarathi]

Hi,

Thank you for your patience. We understand you pain but this seems like a feature request rather than a bug given that IAM roles are always ignored in Local Mode, and pipelines behaviour matches what other SageMaker resources do. This feature needs to be implemented across all the SageMaker resources that support local mode otherwise pipelines behaviour will be different than other resources. We will need to scope the efforts needed and prioritize this accordingly.