amazon-ecs-agent: Agent fails to start ("unable to get vpc id from instance metadata")

Summary

I am attempting to add container instances to an existing cluster. The instances never join the cluster. The ECS agent logs indicate a 404 when trying to fetch the VPC ID from the metadata service. However, these instances were not launched in a VPC and reside in EC2-Classic.

Description

I’ve tried the following AMIs:

amzn-ami-2018.03.m-amazon-ecs-optimized (ami-0796380bc6e51157f)
amzn2-ami-ecs-hvm-2.0.20190204-x86_64-ebs (ami-032564940f9afd5c0)

My /etc/ecs/ecs.config contains this:

ECS_CLUSTER=thunder
ECS_LOGLEVEL=debug

Expected Behavior

Agent starts, and instances join cluster.

Observed Behavior

Agent never starts; instances do not join cluster. Agent logs contain the following:

2019-02-14T23:29:46Z [INFO] Amazon ECS agent Version: 1.25.2, Commit: 0821fbc7
2019-02-14T23:29:46Z [DEBUG] Loaded config: Cluster: thunder,  Region: us-east-1,  DataDir: /data, Checkpoint: true, AuthType: , UpdatesEnabled: true, DisableMetrics: false, PollMetrics: false, PollingMetricsWaitDuration: 15s, ReservedMem: 0, TaskCleanupWaitDuration: 3h0m0s, DockerStopTimeout: 30s, ContainerStartTimeout: 3m0s, TaskCPUMemLimit: 3, , PauseContainerImageName: amazon/amazon-ecs-pause, PauseContainerTag: 0.1.0
2019-02-14T23:29:46Z [INFO] Creating root ecs cgroup: /ecs
2019-02-14T23:29:46Z [INFO] Creating cgroup /ecs
2019-02-14T23:29:46Z [INFO] Loading state! module="statemanager"
2019-02-14T23:29:46Z [INFO] Event stream ContainerChange start listening...
2019-02-14T23:29:46Z [CRITICAL] Unable to initialize Task ENI dependencies: unable to get vpc id from instance metadata: EC2MetadataError: failed to make EC2Metadata request
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>

Environment Details

Instance is in EC2-Classic, with public IP. Hence unclear why fetching VPC ID.
IAM role is ecsInstanceRole with a single AWS-managed policy: AmazonEC2ContainerServiceforEC2Role

Supporting Log Snippets

See below (zip format to make GitHub happy)

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15 (7 by maintainers)

Most upvoted comments

Hey @kian

I did a quick dive into the problem and it looks like there’s a bug in how agent detects classic ec2 instances. This really should degrade gracefully instead of failing like this – and thats something we will need to fix on our end.

That said, you may be able to avoid this code path by adding the following to your ecs.config file:

ECS_ENABLE_TASK_ENI=false

petderek on Feb 15, 2019

@kian You should not need any special setup.

@suneyz This sounds like a bug in the ECS agent. @kian is running in EC2-Classic, which means the instance is not running inside a VPC. The agent should tolerate the lack of a VPC ID and disable features that depend on it (like awsvpc network mode).

samuelkarp on Feb 15, 2019