Shelling into a running ECS task

With containers on ECS there's no server to SSH into, and on Fargate there's no host at all. When I need to look inside a running container (check a config file, run a framework's console command, test a connection from where the app sits) the answer is ECS Exec. It uses Systems Manager Session Manager under the hood, so there are no ports to open and no SSH keys to manage.

The command itself is short. Getting to the point where it works is the longer part.

Prerequisites

1. The Session Manager plugin on your machine. The AWS CLI hands the session to this plugin, and without it you get an error telling you to install it. It's a separate download from the AWS CLI; install it for your OS from the Session Manager documentation and check with:

session-manager-plugin --version

Use AWS CLI v2.

2. Permissions on the task role. The SSM agent runs inside the container and has to open a channel back to Systems Manager. It does that with the task role (taskRoleArn), not the task execution role that pulls images and writes logs. Add this to the task role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssmmessages:CreateControlChannel",
        "ssmmessages:CreateDataChannel",
        "ssmmessages:OpenControlChannel",
        "ssmmessages:OpenDataChannel"
      ],
      "Resource": "*"
    }
  ]
}

The tasks also need a network path to the Systems Manager endpoints: a NAT gateway, a public IP, or a VPC interface endpoint for ssmmessages in private subnets with no internet route.

Your own IAM user or role needs ecs:ExecuteCommand on the cluster or task, plus ecs:DescribeTasks and ecs:ListTasks to find it.

3. Exec enabled on the service. It's off by default, and it's a property of each task, set when the task starts. Turning it on for the service only affects new tasks, so force a deployment to replace the running ones:

aws ecs update-service \
  --cluster example-cluster \
  --service example-service \
  --enable-execute-command \
  --force-new-deployment \
  --region eu-west-2 \
  --profile example

In CloudFormation, CDK or Terraform it's EnableExecuteCommand / enable_execute_command on the service. If you only set it by hand, the next infrastructure deploy turns it off again.

4. A shell in the image. ECS Exec runs a command inside your container. If the image has no /bin/sh, as with distroless or scratch images, there's nothing to run.

On Fargate the tasks need platform version 1.4.0 or later; on EC2 the container instances need a reasonably recent ECS agent. Anything set up in the last few years is fine.

Finding the task

List the running tasks for the service:

aws ecs list-tasks \
  --cluster example-cluster \
  --service-name example-service \
  --region eu-west-2 \
  --profile example

Task ARNs look like arn:aws:ecs:eu-west-2:123456789012:task/example-cluster/<task-id>. You can pass either the full ARN or just the ID at the end. To grab the first one into a variable:

ECS_TASK=$(aws ecs list-tasks \
  --cluster example-cluster \
  --service-name example-service \
  --query 'taskArns[0]' --output text \
  --region eu-west-2 --profile example)

Check exec is actually enabled on that task and the agent is running:

aws ecs describe-tasks \
  --cluster example-cluster \
  --tasks "$ECS_TASK" \
  --query 'tasks[0].{exec:enableExecuteCommand,agents:containers[].managedAgents[].lastStatus}' \
  --region eu-west-2 --profile example

You want exec to be true and the agent status to be RUNNING.

Opening the shell

aws ecs execute-command \
  --cluster example-cluster \
  --task "$ECS_TASK" \
  --container app \
  --interactive \
  --command "/bin/sh" \
  --region eu-west-2 \
  --profile example

--container is the container name from the task definition; you only need it when the task has more than one container. --interactive is required, as interactive sessions are the only kind it supports. Use /bin/bash if the image has it.

Leave with exit. The session also closes after about 20 minutes of no input.

When it fails

The errors are not always helpful. The usual causes, in the order I check them:

• "The execute command failed because execute command was not enabled when the task was run": prerequisite 3. Enable it and force a new deployment.
• "SessionManagerPlugin is not found": prerequisite 1.
• The command hangs, or the agent never reaches RUNNING: the task role is missing the ssmmessages permissions, or the task can't reach the endpoints.
• An AccessDeniedException straight away: your own credentials lack ecs:ExecuteCommand, or you're using the wrong --profile.

AWS publish a shell script, amazon-ecs-exec-checker, in their aws-containers GitHub organisation. Point it at a cluster and task and it checks all of the above and tells you which one is missing.

Be careful in production

A shell inside a production container is as powerful as the application's own credentials. ECS Exec sessions are recorded in CloudTrail as ExecuteCommand calls, and you can configure the cluster to log the session output to CloudWatch Logs or S3. Turn that on for production clusters, and keep ecs:ExecuteCommand to the few people who need it. Anything you change in a container is lost when the task is replaced, so fix the image, not the container.