Skip to content

Generate an ECS execution role for each container #201

Closed

Description

@ale210

Dependency

Overview

We need the container module to generate an ECS execution role for each container it creates, because all ten containers currently share a single hardcoded role. A shared role cannot be scoped to one project's resources, which is what the follow-on issue in this epic needs to do.

One role per container, not one per project — decided 2026-09-01 (Alex). The reason is developer ergonomics and symmetry: the module already generates a task role per container (aws_iam_role.instance), so generating the execution role alongside it means a project team adds a container and gets both roles automatically, with nothing to wire by hand and no shared object to reason about. A per-project role would have to be created elsewhere and passed in.

Terminology, because ECS overloads the word: "container" here means one invocation of the container module — one service and its task definition. It does not mean an ECS container instance, which is the EC2 host registered to the cluster and is a different resource entirely (see ecs-ec2-role).

Action Items

  • Establish what the execution role genuinely needs in order to start a task: pulling the image from its project's ECR repository, writing to its own CloudWatch log group, and reading its project's SSM parameters. Nothing else.
  • Replace the hardcoded execution_role_arn in terraform/modules/container/main.tf with a role created by the module, in the same shape and naming scheme as the aws_iam_role.instance task role immediately above it. Ten containers means ten execution roles.
  • Tag each new role project = var.project_name at creation. The roles are per container but the tag is the project name, so several roles legitimately share a tag value — that is expected, and matches how the ten per-container task roles are tagged.
  • Attach AmazonECSTaskExecutionRolePolicy or a scoped equivalent. Do not carry AmazonSSMFullAccess forward — narrowing it is the follow-on issue in this epic, and reproducing it here would defeat that.
  • Roll this out one service at a time and confirm each task still starts. An execution role that cannot pull an image or read a secret fails the task before the container runs, which surfaces as an application fault rather than a permissions one — so a broken rollout is easy to misdiagnose.
  • Decide what happens to the now-unused shared role incubator-prod-ecs-task-role, and to its stale Terragrunt-era tags terraform_managed = true and last_changed = Sat 2021-May-08. Do not delete the role in this issue — record the decision and raise it separately once nothing references it.
  • After the PR merges, confirm every service is running its expected task count and that each task definition names its own project's execution role.

Resources/Instructions

  • The hardcoded ARN is arn:aws:iam::035866691871:role/incubator-prod-ecs-task-role, set as execution_role_arn in terraform/modules/container/main.tf. Line numbers were accurate when this issue was written (2026-09-01) and may drift; find the code by the resource name if so.
  • Naming trap. The shared role is named incubator-prod-ecs-task-role but is used as the execution role. The actual task roles are named ecs-container-*. The names are inverted relative to their function.
  • The shared role has exactly two attachments today: AmazonECSTaskExecutionRolePolicy and AmazonSSMFullAccess.

Activity

  1. added this to the 02 - Security milestone on Sep 2, 2026
  2. changed the title [-]Generate a per-project ECS execution role[/-] [+]Generate an ECS execution role for each container[/+] on Sep 2, 2026
  3. moved this from New Issue Review to Prioritized Backlog in CoP: DevOps: Project Boardon Sep 5, 2026
  4. self-assigned this
    on Sep 13, 2026
  5. moved this from Prioritized Backlog to In progress (actively working) in CoP: DevOps: Project Boardon Sep 13, 2026
  6. ale210 commented on Sep 13, 2026

    @ale210
    MemberAuthor

    Scope change, agreed 2026-09-13: this issue now also delivers #203. The new execution role's policy is project-scoped from the start rather than a copy of AmazonECSTaskExecutionRolePolicy:

    Access Actions Scope
    ECR auth ecr:GetAuthorizationToken * (not scopable)
    ECR pull ecr:BatchGetImage, ecr:GetDownloadUrlForLayer, ecr:BatchCheckLayerAvailability aws:ResourceTag/project = the project
    Logs logs:CreateLogStream, logs:PutLogEvents the container's own /ecs/<project>-<type>-<env> log group
    Secrets ssm:GetParameters parameter/<project>/*

    ECR is scoped by tag because repository names don't follow the project name (homeuniteus). All 7 repositories carry the correct tag, verified today.

    Delivered in three PRs instead of rolling out one service at a time by hand:

    1. Create the roles and policies behind a temporary module variable, with every service left on the shared role. Check each new role with simulate-principal-policy.
    2. Switch vrms-frontend-dev only, and confirm its task starts.
    3. Switch the other nine and remove the variable.
  7. moved this from In progress (actively working) to Done in CoP: DevOps: Project Boardon Sep 13, 2026
  8. ale210 commented on Sep 13, 2026

    @ale210
    MemberAuthor

    Delivered in #238 (roles and policies), #239 (canary on vrms-frontend-dev) and #240 (the new role became the default). All three are applied.

    Verified 2026-09-13 after #240 applied. All 10 services are running 1/1 on a task definition revision whose execution role is their own ecs-execution-<family>. Every target group target is healthy. Every running task has written to a fresh log stream in its own log group. No stopped task shows ResourceInitializationError or CannotPullContainer.

    Before rollout, the IAM policy simulator confirmed each role can pull only its own project's images and read only its own project's parameters. It denied a repository tagged with another project, another project's parameter, and a lookalike path (/<project>-other/). The simulator can't evaluate CloudWatch Logs ARNs containing /, so the log permission was proven by the canary's live task instead.

    The shared incubator-prod-ecs-task-role is no longer used by any service. It is not deleted: use_own_execution_role = false still falls back to it. Decision: delete the role and that variable together once nothing uses the role, as a separate change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions