docker-py: Cannot pull images from AWS ECR (login does not seem to work properly)

Okay, I ran some experiments the last few days. Had to deal with the 12 hour AWS ECR ticket so it took a little longer to do.

It does seem that there is an issue with docker-py.

Based on my findings, I can either use boto3 or run a sub-process calling the command line to aws ecr. However the only permutation that seems to work with the following steps.

1. use a sub-process to perform the docker login. This will result in the config.json file being updated (not sure if this has any relevance at all or not).
2. Create the docker client via docker_client = docker.from_env(). I have found doing this prior to the sub-process results in it not working properly (unless you have an already valid config.json
3. Then call docker_client.login(username=username, password=password, registry=registry_url)

Whether or not this is expected or not or if I’m doing something wrong, I don’t know. This is what I’ve come up with as steps that work.

This seems like a significant bug that should at least be documented! Ideally, login would throw an exception instead of silently doing nothing. The combination of removing https:// (@mikeage) and setting reauth=True (@dugarsumit) solved the issue for me. Either of those on its own didn’t change the behavior at all.

Looks like this is still an issue? My workaround has been to wrap the whole thing in a context manager that clears out the config.json first, then does the work, and finally returns it to the original state. It’s not pretty but it does the job. (updated to use the decorator)

from contextlib import contextmanager

    @contextmanager
    def _flush_existing_login(registry: str) -> None:
        """ handles the known bug 
            where existing stale creds cause login
            to fail.
            https://github.com/docker/docker-py/issues/2256
        """
        config = Path(Path.home() / ".docker" / "config.json") 
        original = config.read_text()
        as_json = json.loads(original)
        as_json['auths'].pop(registry, None)
        config.write_text(json.dumps(as_json))
        try:
            yield
        finally:
            config.write_text(original)

the docker.login() response usually has a Status: Login Succeeded when it works, but when the user has a ~/.docker/config.json then the reply will be a totally different payload, containing the username/password (AWS and the super long b64 string use for docker login) instead of actually performing the login.

2 of my collegues had that issue, and after deleting the config.json the response was appropriate again.

I too recently ran into this issue, but I don’t think it’s a docker-py bug per se. I discovered the following after digging through the docker source code (written in Go).

If you refer to the Docker API documentation the /auth endpoint actually only tests that registry credentials are valid, and optionally returns a token if the tested registry responds with one for future authentication. There is nothing about credential storage because the daemon does not, in fact, store credentials ever. Any credential storage that takes place is handled by the docker binary itself externally to the API.

When executing docker login on the command line your config.json file is checked for any configured credential store; whatever string is found there is appended to docker-credential- and executed. The Docker project maintains a repo for the most common helpers: https://github.com/docker/docker-credential-helpers

I was specifically using this library for integration with ECR as well; my process now looks like this:

1. Use boto3 ECR client to get Docker credentials
2. Validate the credentials with the docker-py function login(), if desired
3. Use the subprocess Python standard library to call docker-credential-<some configured helper> and pass the ECR credentials on stdin

That being said, I think the docker-py documentation should be updated to call out this fact, and could be patched with credential storage similar to the CLI binary. The way it’s currently described implies similar functionality to docker login on the command line, which is simply not the case.

I’m not sure if this is related to your issue, but I’m having the same problem, and I was able to resolve it by calling docker_client.login with registry equal to registry_url = token['authorizationData'][0]['proxyEndpoint'].replace("https://", ""). Note that I’m removing the leading https://

I’ve done the https:// removal as well as the reauth=True but I’m still running into issues. I have a python service I’m deploying to ECS. The heart of the service is:

        docker_client = docker.from_env()
        username, password = ecr.credentials()
        registry = repo.registry_uri().replace("https://", "")
        docker_client.login(
            username=username, password=password, registry=registry, reauth=True
        )
        docker_client.images.pull(repository=repository, tag=tag)

On my local mac, I can run the above in Pycharm, works great. I can take the same script an bundle it in a container and run the container on my local machine, works great. I then push the container to ECR, and launch the container in ECS. I try and pull the image and I get the dreaded

ERROR: 404 Client Error for http+docker://localhost/v1.40/images/xxx.dkr.ecr.rrr.amazonaws.com/python:3.6.8/json: Not Found ("no such image: xxx.dkr.ecr.rrr.amazonaws.com/python:3.6.8

Ah, found the problem. I was missing ecr:GetDownloadUrlForLayer in the policy for the task.