amazon-ecs-local-container-endpoints: NoCredentialProviders: no valid providers in chain exception

Hi there, can someone please help me understand what I’m doing wrong. I’m following the docs to make local env setup working.

Here is my override file

version: "2"
networks:
  credentials_network:
    driver: bridge
    ipam:
      config:
        - subnet: "169.254.170.0/24"
          gateway: 169.254.170.1
services:
  ecs-local-endpoints:
    image: amazon/amazon-ecs-local-container-endpoints
    volumes:
      - /var/run:/var/run
      - "~/.aws:/home/.aws:ro"
    # env_file:
    #   - ~/.clokta/fusion.env
    environment:
      HOME:
      AWS_PROFILE: "fusion"
    networks:
      credentials_network:
        ipv4_address: "169.254.170.2"
  proxy:
    depends_on:
      - ecs-local-endpoints
    networks:
      credentials_network:
        ipv4_address: "169.254.170.3"
    environment:
      AWS_DEFAULT_REGION: "us-east-1"
      AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/role/blah-role"
      ECS_CONTAINER_METADATA_URI: "http://169.254.170.2/v3"

I’ve logged in through SSO and have STS issued credentials. With this set up I’m making the following request:

GET 169.254.170.2/role/blah-role HTTP/1.1
Host:169.254.170.2
Connection: close

As a result, I’m getting the following response:

HTTP/1.1 500 Internal Server Error
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 25 Feb 2020 19:06:08 GMT
Content-Length: 435
Connection: close

Internal Server Error: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, fusion.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused

ecs local container logs has this line:

ecs-local-endpoints_1  | time="2020-02-25T19:06:07Z" level=error msg="HTTP 500 - NoCredentialProviders: no valid providers in chain\ncaused by: EnvAccessKeyNotFound: failed to find credentials in the environment.\nSharedCredsLoad: failed to load profile, fusion.\nEC2RoleRequestError: no EC2 instance role found\ncaused by: RequestError: send request failed\ncaused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused"

However, if I uncomment

    # env_file:
    #   - ~/.clokta/fusion.env

and rerun docker-compose, everything seems fine.

~/.clokta/fusion.env contains AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. Those values match the [fusion] section of the ~/.aws/credentials which I’m mapping to the container, except aws_access_key_id key and friends are in the lower case.

Is there something I’m missing in the setup?

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 3
  • Comments: 27 (11 by maintainers)

Most upvoted comments

I am getting the same issue HTTP 500 - NoCredentialProviders: no valid providers in chain\ncaused by: EnvAccessKeyNotFound: failed to find credentials in the environment.\nSharedCredsLoad: failed to load profile, spm-dev.\nEC2RoleRequestError: no EC2 instance role found\ncaused by: RequestError: send request failed\ncaused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused, with AWS SSO can’t work out why

Found the problem now. This flat does not work with SSO. It only works when I copy the credentials including the session token into the credentials file. Will raise new ticket

So local container endpoints works for me, and as you noted, it works on Arch Linux. I don’t think there’s a bug in the container. There’s likely something with your setup on Mac that isn’t working. I’m sorry but you’ll probably have to figure that out yourself- the above are the only ideas I can think of to try.