dvc: s3 remote: unexpected error with session token

Bug Report

push: unexpected error encountered

Description

When using a session token for aws s3 remote the dvc push produces the following error: ERROR: unexpected error - An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

Reproduce

Configure ~/aws/credentials with access_key, secret_access_key and session_token; profile name: myprofile
dvc remote add -d myremote s3://mybucket/path
dvc remote modify myremote profile myprofile
dvc add test.txt
dvc push

Alternative:

Configure an aws sso profile named myprofile
aws sso login --profile myprofile
dvc remote add -d myremote s3://mybucket/path
dvc remote modify myremote profile myprofile
dvc add test.txt
dvc push

Expected

test.txt is pushed to the S3 bucket

Environment information

Output of dvc version:

$ dvc version
DVC version: 1.11.7 (brew)
---------------------------------
Platform: Python 3.9.1 on macOS-11.0.1-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Caches: local
Remotes: s3
Repo: dvc, git

Additional Information (if any):

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 2
Comments: 20 (9 by maintainers)

Commits related to this issue

Amazon S3: change parsing of path Fixes #5130 — committed to Christoph-1/dvc by Christoph-1 3 years ago

Most upvoted comments

@Christoph-1 really glad to see this issue has been resolved for you. Don’t hesitate to let us know if you experience any other issue through discord / github.

isidentical on Jun 29, 2021

cache is just an arbitrary prefix, you can do your/original/path/apple etc. and DVC wil use your/original/path/cache/<xx>/<yy> instead of your/original/path/<xx>/<yy> which could have helped in the case of trailing slash. Looking into it.

isidentical on Jun 28, 2021

Also get a similar problem on DVC 2.3.0

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: No AWSAccessKey was presented.

I’ve confirmed I do have access to the S3 bucket and everything works normally on DVC 1

amin-nejad on Jun 21, 2021

Recently #5240 landed, maybe it can help?

isidentical on Jan 13, 2021

Maybe try setting env variables do the job:

export AWS_SECRET_ACCESS_KEY=<> export AWS_ACCESS_KEY_ID=<> export AWS_SESSION_TOKEN=<>

I tried this, but unfortunately it didn’t work. As we’re using SSO, we cannot export env variables. https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-set

Christoph-1 on Jan 7, 2021