dvc: push: Infinite loop in credential configuration detected

Bug Report

Description

I am using dvc with an AWS S3 remote. The credentials to access to S3 are handled by aws-vault.

When using dvc 2.5.4, everything works fine. However, with 2.6.x or newer, I get the following error when I try to dvc push:

ERROR: unexpected error - Infinite loop in credential configuration detected. Attempting to load from profile my-profile-a which has already been visited. Visited profiles: ['my-profile-b', 'my-profile-a']

The error appears if:

  • I try to dvc push after having updated a file inside a dvc-tracked folder.

However, I can dvc push normally if:

  • I add a new file to a dvc-tracked folder.
  • I add a new dvc-tracked file.
  • I update a dvc-tracked file.

My aws-vault credentials configuration is as follows:

[profile my-profile-a]
region=eu-west-3
s3.multipart_threshold = 5GB
credential_process=aws-vault exec --no-session --json my-profile-a
s3=
  multipart_threshold = 5GB

[profile my-profile-b]
region=us-west-1
source_profile = my-profile-a
role_arn = arn:aws:iam::123456789012:role/MyRole
role_session_name = my.name
s3=
  multipart_threshold = 5GB

The dvc config is as follows:

[core]
    remote = s3remote
['remote "s3remote"']
    url = s3://my-bucket
    profile = my-profile-b

A possible explanation is that dvc calls the botocore function that checks the credentials more than once. At first I thought it to be related to this botocore issue but as the error appears only when trying to dvc push a folder after an update to one of its files, the issue seems to be related to dvc.

Reproduce

  • Set aws-vault with a similar configuration
  • Set dvc with a similar configuration
  • Add and push a folder to dvc
  • Update an existing file inside this folder
  • Try to dvc push the folder

Expected

dvc push should be possible.

Environment information

Output of dvc doctor:

$ dvc doctor

DVC version: 3.30.1 (pip)
-------------------------
Platform: Python 3.10.10 on Linux-5.15.0-89-generic-x86_64-with-glibc2.31
Subprojects:
	dvc_data = 2.22.0
	dvc_objects = 1.2.0
	dvc_render = 0.6.0
	dvc_task = 0.3.0
	scmrepo = 1.5.0
Supports:
	http (aiohttp = 3.9.0, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.9.0, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2023.10.0, boto3 = 1.28.64)
Config:
    Global: /home/myname/.config/dvc
    System: /etc/xdg/xdg-ubuntu/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/vgubuntu-root
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/mapper/vgubuntu-root
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/8767037a1b741787f352cb10a11ee443

Additional Information (if any):

$ dvc push -v

2023-12-07 16:52:31,294 DEBUG: v3.30.1 (pip), CPython 3.10.10 on Linux-5.15.0-89-generic-x86_64-with-glibc2.31
2023-12-07 16:52:31,294 DEBUG: command: /home/myname/work/tmp/test-dvc-versions/.venv/bin/dvc push -v
2023-12-07 16:52:31,618 DEBUG: Preparing to transfer data from '/home/myname/work/tmp/test-dvc-versions/.dvc/cache/files/md5' to 's3://my-bucket/myname/test-dvc-versions/files/md5'
2023-12-07 16:52:31,619 DEBUG: Preparing to collect status from 'my-bucket/myname/test-dvc-versions/files/md5'           
2023-12-07 16:52:31,619 DEBUG: Collecting status from 'my-bucket/myname/test-dvc-versions/files/md5'                     
2023-12-07 16:52:31,622 DEBUG: Querying 2 oids via object_exists                                                                         
2023-12-07 16:52:33,528 ERROR: unexpected error - Infinite loop in credential configuration detected. Attempting to load from profile my-profile-a which has already been visited. Visited profiles: ['my-profile-b', 'my-profile-a']                                              
Traceback (most recent call last):
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/dvc/cli/__init__.py", line 211, in main
    ret = cmd.do_run()
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/dvc/cli/command.py", line 27, in do_run
    return self.run()
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/dvc/commands/data_sync.py", line 64, in run
    processed_files_count = self.repo.push(
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/dvc/repo/__init__.py", line 60, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/dvc/repo/push.py", line 117, in push
    push_transferred, push_failed = ipush(
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/dvc_data/index/push.py", line 68, in push
    result = transfer(
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/dvc_data/hashfile/transfer.py", line 204, in transfer
    status = compare_status(
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/dvc_data/hashfile/status.py", line 176, in compare_status
    dest_exists, dest_missing = status(
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/dvc_data/hashfile/status.py", line 136, in status
    exists = hashes.intersection(
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/dvc_data/hashfile/status.py", line 45, in _indexed_dir_hashes
    indexed_dir_exists.update(hashes)
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/tqdm/std.py", line 1182, in __iter__
    for obj in iterable:
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/dvc_objects/db.py", line 359, in list_oids_exists
    in_remote = self.fs.exists(paths, batch_size=jobs)
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 371, in exists
    return fut.result()
  File "/home/myname/.pyenv/versions/3.10.10/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/myname/.pyenv/versions/3.10.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/dvc_objects/executors.py", line 134, in batch_coros
    result = fut.result()
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/s3fs/core.py", line 1035, in _exists
    await self._info(path, bucket, key, version_id=version_id)
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/s3fs/core.py", line 1302, in _info
    out = await self._call_s3(
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/s3fs/core.py", line 341, in _call_s3
    await self.set_session()
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/s3fs/core.py", line 522, in set_session
    self._s3 = await s3creator.get_client()
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/s3fs/utils.py", line 72, in get_client
    self._client = await self._stack.enter_async_context(
  File "/home/myname/.pyenv/versions/3.10.10/lib/python3.10/contextlib.py", line 619, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/aiobotocore/session.py", line 27, in __aenter__
    self._client = await self._coro
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/aiobotocore/session.py", line 173, in _create_client
    credentials = await self.get_credentials()
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/aiobotocore/session.py", line 83, in get_credentials
    self._credentials = await (
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/aiobotocore/credentials.py", line 960, in load_credentials
    creds = await provider.load()
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/aiobotocore/credentials.py", line 689, in load
    return await self._load_creds_via_assume_role(self._profile_name)
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/aiobotocore/credentials.py", line 692, in _load_creds_via_assume_role
    role_config = self._get_role_config(profile_name)
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/botocore/credentials.py", line 1562, in _get_role_config
    self._validate_source_profile(profile_name, source_profile)
  File "/home/myname/work/tmp/test-dvc-versions/.venv/lib/python3.10/site-packages/botocore/credentials.py", line 1613, in _validate_source_profile
    raise InfiniteLoopConfigError(
botocore.exceptions.InfiniteLoopConfigError: Infinite loop in credential configuration detected. Attempting to load from profile my-profile-a which has already been visited. Visited profiles: ['my-profile-b', 'my-profile-a']

2023-12-07 16:52:33,621 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-12-07 16:52:33,622 DEBUG: Removing '/home/myname/work/tmp/.iKooz7Cn9gqk4S6XK6HrH2.tmp'
2023-12-07 16:52:33,622 DEBUG: Removing '/home/myname/work/tmp/.iKooz7Cn9gqk4S6XK6HrH2.tmp'
2023-12-07 16:52:33,623 DEBUG: Removing '/home/myname/work/tmp/.iKooz7Cn9gqk4S6XK6HrH2.tmp'
2023-12-07 16:52:33,624 DEBUG: Removing '/home/myname/work/tmp/test-dvc-versions/.dvc/cache/files/md5/.R58Taoujw2rofT9vbPZxxm.tmp'
2023-12-07 16:52:33,646 DEBUG: Version info for developers:
DVC version: 3.30.1 (pip)
-------------------------
Platform: Python 3.10.10 on Linux-5.15.0-89-generic-x86_64-with-glibc2.31
Subprojects:
	dvc_data = 2.22.0
	dvc_objects = 1.2.0
	dvc_render = 0.6.0
	dvc_task = 0.3.0
	scmrepo = 1.5.0
Supports:
	http (aiohttp = 3.9.0, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.9.0, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2023.10.0, boto3 = 1.28.64)
Config:
	Global: /home/myname/.config/dvc
	System: /etc/xdg/xdg-ubuntu/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/vgubuntu-root
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/mapper/vgubuntu-root
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/8767037a1b741787f352cb10a11ee443

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-12-07 16:52:33,649 DEBUG: Analytics is enabled.
2023-12-07 16:52:33,716 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmpfw6iwkr3', '-v']
2023-12-07 16:52:33,722 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmpfw6iwkr3', '-v'] with pid 29059

About this issue

  • Original URL
  • State: closed
  • Created 7 months ago
  • Comments: 17 (6 by maintainers)

Most upvoted comments

Thanks a lot !

+1
nisace on Mar 4, 2024

Yes it does !

+1
nisace on Mar 4, 2024

@nisace Sorry, I still didn’t have time to look into this. But regarding pull, it is meant to catch errors and fetch whatever it can and then checkout whatever it can and checkout is where previously occured errors might show up as missing files. I’m not sure how this cold cause “infinite loop in credential” error though.

+1
efiop on Jan 24, 2024

@efiop, I’ve looked at it again and noticed that dvc status -c always hit the raise of InfiniteLoopConfigError at this line of botocore (both with or without a dvc cache). However, the expection is effectively raised (meaning the dvc status -c command breaks) only when a dvc cache is present. Thus it seems that the exception get caught somewhere. But I haven’t found where yet. Do you have any clue? Thanks !

+1
nisace on Jan 11, 2024

@efiop thanks for you response ! Yes, fresh push does not produce the error. I’ve tried again and here is the behavior:

  • dvc add a new folder: ok
  • dvc push the folder: ok
  • add a new file to the folder and dvc add: ok
  • dvc status: ok (Data and pipelines are up to date)
  • dvc status -c: ok, produces the following output
        new:                data/folder/file-2.md                                                                                        
	new:                data/folder
  • dvc push the folder: ok
  • dvc push again without any modification: the error appears
  • dvc status: ok (Data and pipelines are up to date)
  • dvc status -c: the error appears

Thus, in fact, there is no need to update a file in the folder to make the error appear.

+1
nisace on Dec 11, 2023
Read more comments on GitHub
← dvc: pull: Pulling a `.dvc` file with the option `-d` produces an AssertionError
dvc: push: Unnotified error when pushing data into HTTP remote →