amazon-ecs-agent: ECS agent stops tasks with "failed to pull container" when it's already successfully pulled the image.

We’ve been seeing an issue recently where the ECS agent will stop tasks with the error “CannotPullContainerError”:

Details
Status reason   CannotPullContainerError: Error: image deathstar:9ce7d67bf402b337a75c8acf49c11881be2f16bd not found

We’ve dug into this a bit, and it looks like the agent is in fact downloading the image properly, but throw the error anyways:

2016-03-11T17:07:54Z [DEBUG] Pulling image module="TaskEngine" image="604238712147.dkr.ecr.us-east-1.amazonaws.com/deathstar:9ce7d67bf402b3Downloading9c11881be2f16bd" status="Downloading [==================================================>] 157.3 MB/157.3 MB
"
2016-03-11T17:07:54Z [DEBUG] Pulling image module="TaskEngine" image="604238712147.dkr.ecr.us-east-1.amazonaws.com/deathstar:9ce7d67bf402b337a75c8acf49c11881be2f16bd" status="Verifying Checksum
"
2016-03-11T17:07:54Z [DEBUG] Pulling image module="TaskEngine" image="604238712147.dkr.ecr.us-east-1.amazonaws.com/deathstar:9ce7d67bf402b337a75c8acf49c11881be2f16bd" status="Pulling repository 604238712147.dkr.ecr.us-east-1.amazonaws.com/deathstar
"
2016-03-11T17:07:54Z [DEBUG] Pulling image complete module="TaskEngine" image="604238712147.dkr.ecr.us-east-1.amazonaws.com/deathstar:9ce7d67bf402b337a75c8acf49c11881be2f16bd"
2016-03-11T17:07:54Z [DEBUG] Pull completed for image module="TaskEngine" image="604238712147.dkr.ecr.us-east-1.amazonaws.com/deathstar:9ce7d67bf402b337a75c8acf49c11881be2f16bd"
2016-03-11T17:07:54Z [INFO] Error transitioning container module="TaskEngine" task="deathstar_staging:426 arn:aws:ecs:us-east-1:604238712147:task/18596178-a7fa-407c-b95a-2270885d4524, Status: (NONE->RUNNING) Containers: [deathstar_staging (NONE->RUNNING),]" container="deathstar_staging(604238712147.dkr.ecr.us-east-1.amazonaws.com/deathstar:9ce7d67bf402b337a75c8acf49c11881be2f16bd) (NONE->RUNNING)" state="PULLED"
2016-03-11T17:07:54Z [DEBUG] Got container event for task module="TaskEngine" task="deathstar_staging:426 arn:aws:ecs:us-east-1:604238712147:task/18596178-a7fa-407c-b95a-2270885d4524, Status: (NONE->RUNNING) Containers: [deathstar_staging (NONE->RUNNING),]"
2016-03-11T17:07:54Z [DEBUG] Handling container change module="TaskEngine" task="deathstar_staging:426 arn:aws:ecs:us-east-1:604238712147:task/18596178-a7fa-407c-b95a-2270885d4524, Status: (NONE->RUNNING) Containers: [deathstar_staging (NONE->RUNNING),]" change="{container:0xc20835cc60 event:{Status:1 DockerContainerMetadata:{DockerId: ExitCode:<nil> PortBindings:[] Error:{transition:Pull msg:Error: image deathstar:9ce7d67bf402b337a75c8acf49c11881be2f16bd not found} Volumes:map[]}}}"

The agent for this host is still connected, and can reach ECR. In the past, we’ve been able to solve this by terminating the affected host. Not a great long term solution for us, though.

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 16 (4 by maintainers)

Most upvoted comments

Those who stumble here wondering what’s up: don’t forget to include the entire repository hostname in your image name when defining a task 😃 e.g. 123456.dkr.ecr.eu-west-1.amazonaws.com/your-repo/your-image-name

+15

atcol on Jul 23, 2016

I needed to ssh into the machine and clean up old docker images to fix this. Docker keeps every tagged image on the machine and it’ll prevent it from downloading new images.

A solution I’ve found is adding a cron job that cleans up the old/unused docker images.

0 * * * * docker rmi $(docker images -q)

+12

nickromano on Sep 2, 2016

Hi @juanrhenals. I’m also hitting this problem regularly. I’m currently using the newest ECS optimized AMI.

Re-open maybe?

schickling on Jun 28, 2016

Sorry I’m an idiot - I forgot the image tag, it works flawlessly!

Pcummings on Jul 15, 2016

What version of the ECS Agent and Docker are you using?

2016-03-11T17:07:54Z [DEBUG] Pulling image complete module="TaskEngine" image="604238712147.dkr.ecr.us-east-1.amazonaws.com/deathstar:9ce7d67bf402b337a75c8acf49c11881be2f16bd"
2016-03-11T17:07:54Z [DEBUG] Pull completed for image module="TaskEngine" image="604238712147.dkr.ecr.us-east-1.amazonaws.com/deathstar:9ce7d67bf402b337a75c8acf49c11881be2f16bd"

These two lines are confusing and get emitted even when there is an error (see here and here). Can you check the Docker daemon log and see what information it has? We’ve found that the Docker remote API (which is how the Agent communicates with Docker) tends to return less information than the daemon shows in the logs. You may want to turn on debug mode for the Docker daemon with --debug.

samuelkarp on Mar 11, 2016