boto3: Boto3 Session cannot find credentials in the environment, throws exception

Here is my code:

import os

import boto3

print os.environ
session = boto3.Session(region_name='us-east-1')

Here’s the contents of os.environ, which get printed to the screen (with some variables stripped). Note that AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are both present.

{
    'CIRCLE_BUILD_NUM': '93',
    'SSH_ASKPASS': 'false',
    'HOME': '/home/ubuntu',
    'CI_PULL_REQUESTS': '',
    'NVM_CD_FLAGS': '',
    'rvm_trust_rvmrcs_flag': '1',
    'CIRCLE_TEST_REPORTS': '/tmp/circle-junit.Kx9Hk38',
    'rvm_path': '/home/ubuntu/.rvm',
    'CIRCLE_USERNAME': '(stripped)',
    'RUBY_VERSION': 'ruby-1.9.3-p448',
    'SSH_CLIENT': '10.16.130.250 40127 22',
    'CIRCLE_REPOSITORY_URL': '(stripped)',
    'USER': 'ubuntu',
    'NVM_DIR': '/home/ubuntu/nvm',
    'M2': '/home/ubuntu/.m2/apache-maven-3.2.5/bin',
    'PATH': '/home/ubuntu/webhooks/venv/bin:/usr/local/heroku/bin:/opt/google-cloud-sdk/bin:/home/ubuntu/webhooks/venv/bin:/home/ubuntu/.pyenv/shims:/home/ubuntu/.pyenv/bin:/home/ubuntu/.local/bin:/home/ubuntu/.go_workspace/bin:/usr/local/go_workspace/bin:/usr/local/go/bin:/opt/ghc/7.6.3/bin:/opt/happy/1.19.3/bin:/opt/alex/3.1.3/bin:/opt/cabal/1.22/bin:/opt/google-cloud-sdk/bin:/home/ubuntu/.m2/apache-maven-3.2.5/bin:/home/ubuntu/nvm/v0.10.33/bin:/home/ubuntu/.phpenv/shims:/home/ubuntu/.phpenv/bin:/home/ubuntu/.rvm/gems/ruby-1.9.3-p448/bin:/home/ubuntu/.rvm/gems/ruby-1.9.3-p448@global/bin:/home/ubuntu/.rvm/rubies/ruby-1.9.3-p448/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/android-sdk-linux/tools:/usr/local/android-sdk-linux/platform-tools:/home/ubuntu/.rvm/bin:/home/ubuntu/.composer/vendor/bin:/usr/local/gradle-1.10/bin:/usr/local/heroku/bin:/home/ubuntu/.rvm/bin',
    'CIRCLE_PROJECT_USERNAME': '(stripped)',
    'PS1': '(venv)',
    'DISPLAY': ':99',
    'MAKEFLAGS': '',
    'CIRCLE_COMPARE_URL': '(stripped)',
    'NVM_BIN': '/home/ubuntu/nvm/v0.10.33/bin',
    'TERM': 'vt100',
    'SHELL': '/bin/bash',
    'CIRCLECI': 'true',
    'XDG_SESSION_COOKIE': '(stripped)',
    'PYENV_SHELL': 'bash',
    'RBENV_SHELL': 'bash',
    'SHLVL': '2',
    '_system_arch': 'x86_64',
    'rvm_gemset_create_on_use_flag': '1',
    'rvm_version': '1.26.10 (1.26.10)',
    '_system_name': 'Ubuntu',
    'MAVEN_OPTS': '-Xmx2048m',
    'NVM_PATH': '/home/ubuntu/nvm/v0.10.33/lib/node',
    'GIT_ASKPASS': 'echo',
    'CIRCLE_NODE_INDEX': '0',
    'CIRCLE_NODE_TOTAL': '1',
    'MANPATH': '/home/ubuntu/nvm/v0.10.33/share/man:/home/ubuntu/.rvm/rubies/ruby-1.9.3-p448/share/man:/usr/local/man:/usr/local/share/man:/usr/share/man:/home/ubuntu/.rvm/man',
    'GOPATH': '/home/ubuntu/.go_workspace:/usr/local/go_workspace',
    'CIRCLE_PREVIOUS_BUILD_NUM': '92',
    'M2_HOME': '/home/ubuntu/.m2/apache-maven-3.2.5',
    'JAVA_HOME': '/usr/lib/jvm/jdk1.7.0',
    'MFLAGS': '',
    'GEM_PATH': '/home/ubuntu/.rvm/gems/ruby-1.9.3-p448:/home/ubuntu/.rvm/gems/ruby-1.9.3-p448@global',
    'rvm_bin_path': '/home/ubuntu/.rvm/bin',
    'LANG': 'en_US.UTF-8',
    'CI': 'true',
    '_system_version': '12.04',
    'rvm_prefix': '/home/ubuntu',
    'LOGNAME': 'ubuntu',
    'CI_REPORTS': '/tmp/circle-junit.Kx9Hk38',
    'CIRCLE_SHA1': 'bd4cbd0943d1cde45abf6cacb580a3a389cc897a',
    'IRBRC': '/home/ubuntu/.rvm/rubies/ruby-1.9.3-p448/.irbrc',
    '_system_type': 'Linux',
    'MY_RUBY_HOME': '/home/ubuntu/.rvm/rubies/ruby-1.9.3-p448',
    'CI_PULL_REQUEST': '',
    'ANDROID_HOME': '/usr/local/android-sdk-linux',
    'AWS_ACCESS_KEY_ID': 'AKIA(stripped)',
    '_': '/home/ubuntu/webhooks/venv/bin/python',
    'ADB_INSTALL_TIMEOUT': '10',
    'CIRCLE_BUILD_URL': '(private)',
    'VIRTUAL_ENV': '/home/ubuntu/webhooks/venv',
    'NVM_IOJS_ORG_MIRROR': 'https://iojs.org/dist',
    'SSH_TTY': '/dev/pts/5',
    'OLDPWD': '/home/ubuntu/webhooks',
    'CIRCLE_PROJECT_REPONAME': 'webhooks',
    'ANDROID_NDK': '/usr/local/android-ndk',
    'CIRCLE_BRANCH': 'develop',
    'MAKELEVEL': '1',
    'PWD': '/home/ubuntu/webhooks/src',
    'CIRCLE_ARTIFACTS': '/tmp/circle-artifacts.Hk67jto',
    'AWS_SECRET_ACCESS_KEY': '(secret)',
    'MAIL': '/var/mail/ubuntu',
    'SSH_CONNECTION': '10.16.130.250 40127 10.0.3.193 22',
    'GEM_HOME': '/home/ubuntu/.rvm/gems/ruby-1.9.3-p448',
    'NVM_NODEJS_ORG_MIRROR': 'https://nodejs.org/dist'
}

However, the next line yields the following stack trace:

Traceback (most recent call last):
  File "deploy.py", line 11, in main
    session = boto3.Session(region_name='us-east-1')
  File "/home/ubuntu/webhooks/src/boto3/boto3/session.py", line 80, in __init__
    self._setup_loader()
  File "/home/ubuntu/webhooks/src/boto3/boto3/session.py", line 112, in _setup_loader
    self._loader = self._session.get_component('data_loader')
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 687, in get_component
    return self._components.get_component(name)
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 868, in get_component
    self._components[name] = factory()
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 176, in <lambda>
    lambda:  create_loader(self.get_config_variable('data_path')))
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 256, in get_config_variable
    elif self._found_in_config_file(methods, var_config):
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 277, in _found_in_config_file
    return var_config[0] in self.get_scoped_config()
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 339, in get_scoped_config
    profile_map = self._build_profile_map()
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 211, in _build_profile_map
    self._profile_map = self.full_config['profiles']
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 366, in full_config
    self._config = botocore.configloader.load_config(config_file)
  File "/home/ubuntu/webhooks/src/botocore/botocore/configloader.py", line 105, in load_config
    parsed = raw_config_parse(config_filename)
  File "/home/ubuntu/webhooks/src/botocore/botocore/configloader.py", line 126, in raw_config_parse
    raise botocore.exceptions.ConfigNotFound(path=path)
botocore.exceptions.ConfigNotFound: The specified config file (/home/ubuntu/.aws/config) could not be found.

I expect that boto3/botocore will be able to find my credentials in the environment. The documentation states they should be load-able from the environment:

Boto3 will check these environment variables for credentials:

AWS_ACCESS_KEY_ID The access key for your AWS account. AWS_SECRET_ACCESS_KEY The secret key for your AWS account. AWS_SESSION_TOKEN The session key for your AWS account. This is only needed when you are using temporary credentials. > The AWS_SECURITY_TOKEN environment variable can also be used, but is only supported for backwards compatibility purposes. AWS_SESSION_TOKEN is supported by multiple AWS SDKs besides python.

I’m confused why this is occurring.

About this issue

Original URL
State: closed
Created 8 years ago
Reactions: 1
Comments: 24 (2 by maintainers)

Most upvoted comments

2021 and still this issue is present. even with aws_ssm plugin which uses boto

Example

- name: lookup ssm parameter store in the current region
  debug: msg="{{ lookup('aws_ssm', '/somekey/subkey') }}"

This will crash due to unauthorised/no permission

Even though I have exported AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to environment as mentioned https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-environment-variables

Only way to work is ugly and specify credentials for every ssm plugin call like:

- name: lookup ssm parameter store in the current region
  debug: msg="{{ lookup('aws_ssm', '/mysuperparam', region=eu-central-1, aws_access_key='verysecretid', aws_secret_key='secreterkey'  ) }}"

Please fix boto 3 so it can read enviornment variables and save us so much boilerplate code

+15

kristijorgji on Apr 14, 2021

I still have this problem quite often. I’m using IAM roles, so I have no credentials file (only a .aws/config). I see it most often during runs of long multiprocessing jobs.

+10

smstec on Feb 21, 2018

I’ve “solved” this problem by creating empty $HOME/.aws/config and $HOME/.aws/credentials files and now getting the following error message:

Traceback (most recent call last):
  File "deploy.py", line 43, in <module>
    main(args.realm)
  File "deploy.py", line 29, in main
    results = [client.update_function_code(FunctionName=arn, ZipFile=data) for arn in arns]
  File "/home/ubuntu/webhooks/src/botocore/botocore/client.py", line 236, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ubuntu/webhooks/src/botocore/botocore/client.py", line 489, in _make_api_call
    operation_model, request_dict)
  File "/home/ubuntu/webhooks/src/botocore/endpoint.py", line 117, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/ubuntu/webhooks/src/botocore/endpoint.py", line 146, in _send_request
    success_response, exception):
  File "/home/ubuntu/webhooks/src/botocore/endpoint.py", line 219, in _needs_retry
    caught_exception=caught_exception)
  File "/home/ubuntu/webhooks/src/botocore/botocore/hooks.py", line 226, in emit
    return self._emit(event_name, kwargs)
  File "/home/ubuntu/webhooks/src/botocore/botocore/hooks.py", line 209, in _emit
    response = handler(**kwargs)
  File "/home/ubuntu/webhooks/src/botocore/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/home/ubuntu/webhooks/src/botocore/botocore/retryhandler.py", line 250, in __call__
    caught_exception)
  File "/home/ubuntu/webhooks/src/botocore/botocore/retryhandler.py", line 265, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/home/ubuntu/webhooks/src/botocore/botocore/retryhandler.py", line 313, in __call__
    caught_exception)
  File "/home/ubuntu/webhooks/src/botocore/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(attempt_number, caught_exception)
  File "/home/ubuntu/webhooks/src/botocore/botocore/retryhandler.py", line 355, in _check_caught_exception
    raise caught_exception
ValueError: You can only send PreparedRequests.

kevinburke on Apr 18, 2016

FWIW, I had this problem and realized that it was because I switched to using Tox, but I neglected to place passenv = * in my tox.ini file.

kernelpanek-segfault on Jun 16, 2016

I recently started having this issue after version 1.7.x

tommyshere on May 18, 2018

I gave up on using anything but $HOME/.aws/config and $HOME/.aws/credentials files. It works both on Windows and Linux for me. Placing the values in environment variables does not work for me.

millerbryan on May 20, 2016

@hobofan : virtualenv are recommended, even for cloud. One issue of python package is access without sudo, virtualenv allow you to do that. Also AWS resources has its own catch.

For aws credentials , the common mistake made for multiple profile is omitting the “profile” in the config

#~/.aws/credential
[default] 
aws_access_key_id= ....
aws_secret_access_key=....

[superapp]
aws_access_key_id= ....
aws_secret_access_key=....

# The catch, 
#~/.aws/config
[default]
region = .....

# so the alternate profile, in the config, you must prefix it with profile , follow by space, then profile name
[profile superapp]
region = .....

Then calling alternate profile should be straightforward as shown in the API

boto3.session.Session( profile_name="superapp")

commutecat on Jun 1, 2016