amazon-ecs-agent: ECS agent opens too many connections to Docker via /var/run/docker.sock

Hi, we faced the problem with too many socket connections to Docker (/var/run/docker.sock) on latest ECS optimized AMI ami-6bb2d67c. We upgraded to it once got message from Amazon quoted below. But we hit exactly the same problem that supposed to be fixed in that AMI.

According to our records, you are currently running or you have recently launched an Amazon ECS-optimized AMI where the agent may open too many socket connections to Docker, which can exhaust the limit of Docker socket connections and make the agent unable to start new tasks (agent versions 1.11.1 and 1.12.0).

We recommend that you update to the latest agent [1] or update to the latest Amazon ECS-optimized AMI

Too many opened connections to docker:

[root@ip-172-31-25-188 ~]# netstat -x | grep '/var/run/docker.sock' | wc -l
930

[root@ip-172-31-25-188 ~]# netstat -x | grep '/var/run/docker.sock' | head -3
unix  3      [ ]         STREAM     CONNECTED     685428 /var/run/docker.sock
unix  3      [ ]         STREAM     CONNECTED     683558 /var/run/docker.sock
unix  2      [ ]         STREAM     CONNECTED     529461 /var/run/docker.sock

As soon as it happens docker ps hangs but containers seems to be continue running but ECS agent can’t stop/start new ones anymore since it hangs too so those tasks get stuck in PENDING state:

[root@ip-172-31-25-188 ~]# curl 127.0.0.1:51678/v1/tasks
{"Tasks":[{"Arn":"arn:aws:ecs:us-east-1:000:task/cd562f5f-2dbb-45e6-818f-138ce28176f9","DesiredStatus":"STOPPED","KnownStatus":"RUNNING","Family":"tf-strings-search-service-prod","Version":"84","Containers":[{"DockerId":"e71a83367120f9727c1bc631be4fd60080c12ea8b18d5454c9772fb1b2896e3d","DockerName":"ecs-tf-strings-search-service-prod-84-logstashshipper-fab2888deceac0804c00","Name":"logstashshipper"},{"DockerId":"18755b2eeaec7a215f4f6beff296e15216c1fc36ed6ee3204fa60d2e711e827e","DockerName":"ecs-tf-strings-search-service-prod-84-search-service-86d780fce4b1eea56700","Name":"search-service"}]},{"Arn":"arn:aws:ecs:us-east-1:000:task/9d602625-3285-4b59-8105-38742aaf8f08","DesiredStatus":"RUNNING","KnownStatus":"PENDING","Family":"tf-strings-search-service-prod","Version":"85","Containers":[]}]}

Please see my comment in another ticket that contains more debugging data like ECS agent stack dump: https://github.com/aws/amazon-ecs-agent/issues/515#issuecomment-244331856