amazon-ecs-agent: ECS agent fails to launch with latest AMI/Docker

We’ve also been encountering this issue, thankfully in a non-production environment. I got the docker daemon running again by manually restarting the instance, and that aloud our cluster to connect & run the needed container.

The instance is spun up by an AutoScalingGroup. Here’s the following output as requested, and below is the user-data for the launch configuration. amazon-ecs-docker-log-errors.txt

user-data:

#!/bin/bash

echo ECS_CLUSTER=[cluster_name] > /etc/ecs/ecs.config

yum install -y docker service docker start usermod -a -G docker ec2-user

Hope this helps!

@alexmac My apologies for the delay in response; I got pretty busy last week and this week with DockerCon.

So, a bit of background on what is happening and how:

When we build the AMI, we include a BlockDeviceMapping for an empty EBS volume. At boot, upstart on the instance starts running various software, including cloud-init. Among other things like setting up SSH using the public key you specified when launching the instance, cloud-init is used to configure the instance on boot. The ECS-optimized AMI specifies some cloud-config configuration in a file located at /etc/cloud/cloud.cfg.d/90_ecs.cfg and tells cloud-init to invoke docker-storage-setup through the cloud-init-per helper as a bootcmd. The cloud-config configuration is read very early in the boot process, prior to Docker being started, and bootcmds in particular are executed early in the boot process (this is different from normal user-data scripts, which are executed toward the end). We picked a bootcmd as it was a good way for us to ensure that docker-storage-setup ran before Docker was started the very first time.

I haven’t used Packer before, but there are a few different general techniques you might be able to apply. For example:

• Option A:
  1. Inject your own cloud-config configuration when the source instance is launched that overrides the bootcmd
  2. Run whatever scripts you need to prepare the instance normally
  3. Clean up Docker (stop Docker, remove /var/lib/docker, and remove /etc/sysconfig/docker-storage)
  4. Shut down the instance and snapshot the root volume
  5. Register an AMI with the root volume snapshot and a BlockDeviceMapping for the second volume (as /dev/xvdcz) without a snapshot
  6. This should give you roughly the same experience as launching an ECS-optimized AMI in that docker-storage-setup should run and set up the second volume as the LVM thin pool
• Option B:
  1. Launch an instance of the normal Amazon Linux AMI
  2. Create a volume from the public snapshot of the root volume of the ECS-optimized AMI and attach it to your instance
  3. Perform whatever modifications you want on that volume, then detach and snapshot
  4. Register an AMI with the that snapshot and a BlockDeviceMapping for the second volume (as /dev/xvdcz) without a snapshot
  5. Again, this should give you roughly the same experience
• Option C: Use your existing process, but specify the size of /dev/xvdcz explicitly at launch through the BlockDeviceMapping parameter of RunInstances and use docker ps to wait for initialization to finish prior to stopping Docker.

I haven’t tested each of these, but hopefully this helps give you some general ideas of how you can approach it.