amazon-ecs-agent: Containers configured with awslogs never get placed on an instance

Summary

When trying to set up logs on Windows containers, setting them to auto-configure for awslogs causes the containers to never be placed. The CloudWatch log group does get created.

Description

Two Windows EC2 instances were set up as part of a cluster using the Windows_Server-2016-English-Full-ECS_Optimized-2018.05.01 (ami-46c77939) image. A service set up with containers that require awslogs can’t place containers because “no container instance met all of its requirements.” Trying to run a taks via the API returns a failure of “ATTRIBUTE”.

Expected Behavior

Tasks are placed on instances and log streams are created and content pushed to it.

Observed Behavior

Tasks never get placed.

Environment Details

Docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 17.06.2-ee-10
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.2214.amd64fre.rs1_release_1.180402-1758)
Operating System: Windows Server 2016 Datacenter
OSType: windows
Architecture: x86_64
CPUs: 2
Total Memory: 4GiB
Name: EC2AMAZ-Q72RKI3
ID: HYRW:725W:EX6L:YSQB:CN5Z:OXVV:4FTS:KXNN:2CK2:VZKY:IBJF:HLDG
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Metadata:

PS C:\Users\Administrator> curl http://localhost:51678/v1/metadata


StatusCode        : 200
StatusDescription : OK
Content           : {"Cluster":"mobiledev-cluster","ContainerInstanceArn":"arn:aws:ecs:us-east-1:169164411397:container-instance/910a2bc4-4972-4dea-8b8b-b4d5e36ca7ed","Version":"Amazon ECS Agent -
                    v1.17.3 (159ae5c3)"}
RawContent        : HTTP/1.1 200 OK
                    Content-Length: 197
                    Content-Type: text/plain; charset=utf-8
                    Date: Wed, 23 May 2018 13:48:44 GMT

                    {"Cluster":"mobiledev-cluster","ContainerInstanceArn":"arn:aws:ecs:us-east-1:16916...
Forms             : {}
Headers           : {[Content-Length, 197], [Content-Type, text/plain; charset=utf-8], [Date, Wed, 23 May 2018 13:48:44 GMT]}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : System.__ComObject
RawContentLength  : 197

One thing I noticed is the task definition requires an attribute of ecs.capability.execution-role-awslogs but when I describe the instance that attribute is not found:

{
    "failures": [],
    "containerInstances": [
        {
            "status": "ACTIVE",
            "registeredAt": 1526920787.49,
            "registeredResources": [
                {
                    "integerValue": 2048,
                    "longValue": 0,
                    "type": "INTEGER",
                    "name": "CPU",
                    "doubleValue": 0.0
                },
                {
                    "integerValue": 4095,
                    "longValue": 0,
                    "type": "INTEGER",
                    "name": "MEMORY",
                    "doubleValue": 0.0
                },
                {
                    "name": "PORTS",
                    "longValue": 0,
                    "doubleValue": 0.0,
                    "stringSetValue": [
                        "135",
                        "445",
                        "3389",
                        "2376",
                        "139",
                        "2375",
                        "80",
                        "5985",
                        "51678",
                        "51679",
                        "53"
                    ],
                    "type": "STRINGSET",
                    "integerValue": 0
                },
                {
                    "name": "PORTS_UDP",
                    "longValue": 0,
                    "doubleValue": 0.0,
                    "stringSetValue": [],
                    "type": "STRINGSET",
                    "integerValue": 0
                }
            ],
            "ec2InstanceId": "[omit]",
            "agentConnected": true,
            "containerInstanceArn": "[omit]",
            "pendingTasksCount": 0,
            "remainingResources": [
                {
                    "integerValue": 1024,
                    "longValue": 0,
                    "type": "INTEGER",
                    "name": "CPU",
                    "doubleValue": 0.0
                },
                {
                    "integerValue": 3071,
                    "longValue": 0,
                    "type": "INTEGER",
                    "name": "MEMORY",
                    "doubleValue": 0.0
                },
                {
                    "name": "PORTS",
                    "longValue": 0,
                    "doubleValue": 0.0,
                    "stringSetValue": [
                        "2375",
                        "8080",
                        "135",
                        "445",
                        "3389",
                        "2376",
                        "139",
                        "80",
                        "5985",
                        "51678",
                        "51679",
                        "19800",
                        "53"
                    ],
                    "type": "STRINGSET",
                    "integerValue": 0
                },
                {
                    "name": "PORTS_UDP",
                    "longValue": 0,
                    "doubleValue": 0.0,
                    "stringSetValue": [],
                    "type": "STRINGSET",
                    "integerValue": 0
                }
            ],
            "version": 342,
            "attributes": [
                {
                    "name": "ecs.ami-id",
                    "value": "ami-46c77939"
                },
                {
                    "name": "com.amazonaws.ecs.capability.logging-driver.json-file"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.30"
                },
                {
                    "name": "ecs.capability.execution-role-ecr-pull"
                },
                {
                    "name": "ecs.capability.container-health-check"
                },
                {
                    "name": "ecs.availability-zone",
                    "value": "us-east-1c"
                },
                {
                    "name": "ecs.instance-type",
                    "value": "t2.medium"
                },
                {
                    "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.24"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.26"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.27"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.28"
                },
                {
                    "name": "com.amazonaws.ecs.capability.privileged-container"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
                },
                {
                    "name": "com.amazonaws.ecs.capability.ecr-auth"
                },
                {
                    "name": "ecs.os-type",
                    "value": "windows"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.20"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.22"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.23"
                },
                {
                    "name": "com.amazonaws.ecs.capability.task-iam-role"
                }
            ],
            "versionInfo": {
                "agentVersion": "1.17.3",
                "agentHash": "159ae5c3",
                "dockerVersion": "DockerVersion: 17.06.2-ee-10"
            },
            "runningTasksCount": 1,
            "attachments": []
        }
    ]
}

Supporting Log Snippets

ECS agent logs: ecs-agent-logs.zip Docker events: docker-events.zip

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 17 (2 by maintainers)

Commits related to this issue

Most upvoted comments

Hi @parkrrr,

I saw you have specified the Task execution role, actually it’s designed for Fargate task, you don’t need to set it if you are using EC2. If you want to use it on EC2, you must set ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE to true when starting the instance, as Agent only register itself once when it initializes. There are two ways to solve your problem:

  1. Remove the Task execution role.

  2. Keep Task execution role, create another two Windows instances, add one more line PowerShell in user data when creating them:

[Environment]::SetEnvironmentVariable("ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE", 
$TRUE, "Machine")

Thanks, Haikuo

+23
haikuoliu on May 25, 2018

if one is using "Installing the Amazon ECS Container Agent on a non-Amazon Linux EC2 Instance " from https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html on Step 9. append /etc/ecs/ecs.config with following

ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true

task execution role is required if using “AWS Systems Manager Parameter Store” for storing sensitive data so it can not be removed in these cases.

+13
kortemik on Feb 27, 2019

@CharlesMichaelReed 's comments helped me the most here. To clarify, set the environment variable before the Initialize-ECSAgent comment and use the string “true”. See below.

<powershell>
  Import-Module ECSTools
  [Environment]::SetEnvironmentVariable("ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE", "true", "Machine")
  Initialize-ECSAgent –Cluster DemoCluster -EnableTaskIAMRole
</powershell>

To save others the time when referencing the above, this should work if the environment is configured prior to the Initialize-ECSAgent call. I also had to update $TRUE to “true” in the current version before the container instance would run.

Hope it helps.

+2
peterjgrainger on Nov 10, 2020

Hello, It is EnableTaskIAMRole , not EnableIAMTaskRole

+2
sharanyad on Mar 11, 2020
Read more comments on GitHub
← amazon-ecs-agent: awslogs-stream option not supported by the awslogs driver
amazon-ecs-agent: CredentialsV2Request: ID not found. Request IP Address: --------- →