dvc: ListObjectsV2 operation: Access Denied

Hi,

I got strange behaviours when upgrading from 2.1.0 to 2.5.0.

On some DVC repo, I get an unexpected error - _find() got an unexpected keyword argument 'prefix' or a forbidden error 403 on other.

My setup is pretty simple, I have an s3 remote with AWS in each DVC repo and I simply add and push some data. Nothing fancy, no pipeline.

Setup:

Ubuntu 20.10 Python 3.8 DVC 2.1.0 or 2.5.0

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 29 (10 by maintainers)

Most upvoted comments

Thanks! Ok, so something more complex is going on then. Will need to take a closer look.

CC @isidentical maybe you’ve run into this before.

efiop on Jul 19, 2021

okay, after some downgrade testing working on 2.1.0. and next version(2.2.0) it stop working

Kavan72 on Jul 19, 2021

standard aws s3

daniele-salerno on Jul 14, 2021

Working with DVC 2.6.3 and s3 updated now.

Will close the issue.

davebulaval on Aug 18, 2021

Btw, @Kavan72 @davebulaval @daniele-salerno Could you try running:
aws s3api list-objects-v2 --bucket my-bucket
? Just making sure you have permission to use listobjectsv2.
What is the expected output? I get a JSON file with a key “contents” and the value is a list of dict with 4 keys/values.

yes that the expected output

Kavan72 on Jul 19, 2021

Btw, @Kavan72 @davebulaval @daniele-salerno Could you try running:
aws s3api list-objects-v2 --bucket my-bucket
? Just making sure you have permissions to use listobjectsv2.

yes working

{
    "Contents": [
        {
            "Key": "pdf_extractor_models/",
            "LastModified": "2021-07-19T10:36:42+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD"
        },
        {
            "Key": "pdf_extractor_models/46/671b32efd11065dd23658f78ba9f09",
            "LastModified": "2021-07-19T12:47:16+00:00",
            "ETag": "\"2f6bf9f23d4469fa07427bb7f0560d0d-2\"",
            "Size": 14385567,
            "StorageClass": "STANDARD"
        }
    ]
}

Kavan72 on Jul 19, 2021

yes, first i’ve installed binary package and for some downgrade testing i removed binary package and installed from pip, but getting same error

Kavan72 on Jul 19, 2021

i’m having same problem

(venv) ➜  pdf_api git:(master) dvc push         
ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden                                                                                                                      
Traceback (most recent call last):
  File "s3fs/core.py", line 246, in _call_s3
  File "aiobotocore/client.py", line 154, in _make_api_call
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "s3fs/core.py", line 1057, in _info
  File "s3fs/core.py", line 970, in _simple_info
  File "s3fs/core.py", line 265, in _call_s3
PermissionError: Access Denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "s3fs/core.py", line 246, in _call_s3
  File "aiobotocore/client.py", line 154, in _make_api_call
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "dvc/main.py", line 55, in main
  File "dvc/command/base.py", line 50, in do_run
  File "dvc/command/data_sync.py", line 57, in run
  File "dvc/repo/__init__.py", line 51, in wrapper
  File "dvc/repo/push.py", line 44, in push
  File "dvc/data_cloud.py", line 79, in push
  File "dvc/remote/base.py", line 57, in wrapper
  File "dvc/remote/base.py", line 494, in push
  File "dvc/remote/base.py", line 351, in _process
  File "dvc/remote/base.py", line 195, in _status
  File "dvc/remote/base.py", line 145, in hashes_exist
  File "dvc/objects/db/base.py", line 438, in hashes_exist
  File "dvc/objects/db/base.py", line 389, in list_hashes_exists
  File "concurrent/futures/_base.py", line 611, in result_iterator
  File "concurrent/futures/_base.py", line 439, in result
  File "concurrent/futures/_base.py", line 388, in __get_result
  File "concurrent/futures/thread.py", line 57, in run
  File "dvc/objects/db/base.py", line 380, in exists_with_progress
  File "dvc/fs/fsspec_wrapper.py", line 92, in exists
  File "fsspec/asyn.py", line 87, in wrapper
  File "fsspec/asyn.py", line 68, in sync
  File "fsspec/asyn.py", line 24, in _runner
  File "s3fs/core.py", line 802, in _exists
  File "s3fs/core.py", line 1061, in _info
  File "s3fs/core.py", line 1004, in _version_aware_info
  File "s3fs/core.py", line 265, in _call_s3
PermissionError: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dvc/__main__.py", line 7, in <module>
  File "dvc/main.py", line 88, in main
  File "dvc/info.py", line 35, in get_dvc_info
  File "dvc/info.py", line 127, in _get_supported_remotes
  File "importlib/metadata.py", line 531, in version
  File "importlib/metadata.py", line 504, in distribution
  File "importlib/metadata.py", line 177, in from_name
importlib.metadata.PackageNotFoundError: adlfs
[20135] Failed to execute script __main__

in my case. i’m a owner of my bucket so there is no perm issue i guess. and also aws ls is working fine

Kavan72 on Jul 19, 2021

@davebulaval Not seeing the prefix error anymore, so looks like the upgrade helped. The current error looks similar to #5130 . Could you try aws s3 ls s3://... with the url of your remote, please?

Don’t know why but somehow that command fixed the issue for me

daniele-salerno on Jul 14, 2021

@davebulaval Not seeing the prefix error anymore, so looks like the upgrade helped. The current error looks similar to https://github.com/iterative/dvc/issues/5130 . Could you try aws s3 ls s3://... with the url of your remote, please?

efiop on Jul 6, 2021