spinnaker: Clouddriver: Intermitent DockerRegistryImageCachingAgent Could not load tags for private ECR

The latest version of clouddriver coudl’t load the ECR image tags and returns the error

WARN 1 --- [ecutionAction-3] .d.r.p.a.DockerRegistryImageCachingAgent : Could not load tags for <IMAGE_NAME> in https://<AWS_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com, reason: Status: 404, Method: GET, URL: https://<AWS_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/v2/<REPOSITORY>/tags/list%3Flast=..., Message: Not Found

I’m using the Docker Registry command configuration (ignore <> format values)

...
dockerRegistry:
      enabled: true
      accounts:
      - name: <name>
        requiredGroupMembership: []
        permissions: {}
        address: https://<AWS_ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com
        username: AWS
        passwordCommand: aws --region us-east-1 ecr get-authorization-token --output
          text --query 'authorizationData[].authorizationToken' | base64 -d | sed
          's/^AWS://'
        email: <FAKE_EMAIL>
        cacheIntervalSeconds: 30
        clientTimeoutMillis: 60000
        cacheThreads: 2
        paginateSize: 1000
        sortTagsByDate: false
        trackDigests: false
        insecureRegistry: false
        repositories: []
      primaryAccount: <name>

cacheThreads is not relevant, I’ve tried with 1.

The AmazonEC2ContainerRegistryReadOnly policy attachment is working properly. All the registry repositories are configured equals, all of those are private and all of those contain tags.

I’ve tried start with a cleaned Redis to avoid the serialization problems.

Spinnaker Version: 1.37.3

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (10 by maintainers)

Most upvoted comments

Update: I am able to reproduce the issue with an ECR account, working on the fix.

com.netflix.spinnaker.kork.retrofit.exceptions.SpinnakerHttpException: Status: 405, Method: GET, URL: https://<AWS_ACC>.dkr.ecr.ap-south-1.amazonaws.com/v2/my-alpine/tags/list?last=Nde5FCDw1zWkZzJ6A1Z62peakkDPEPvmR3NM9aL3q7yzx95U3S6iyeBRKbWaSMthLeZ6FvJmBiVg1Woj%252FNOhjepFq7kPAr%252FEw%252FYxfcJoHPrRtyiqe%252F1Qj5V9ZxkRw9GwXH%252BWwaHjHAnN0A1toOLyBZ9N52%252BloQCrQRkcIFULfkBoIdAEO2yIn88Sy4ASRhZ%252B8XhnF%252BF0DBKDr%252B6xA9qquZewsLyd7NEDx0mFv1EcUjqVHbvg7uDTKtzIDZiclAELtvuSVxxpi56CCCOqkc77YpQSdji3Xmp299otByhHR8r1BJdvMXB2R6H1YeVMd1GvzppUgB6J%252F9D9HLnOKXm%252FZ%252FKx2K7Ea3C8cBwm8KkO%252FVCdTSJf2HXrtgLgHWRSzY4gPQkqlwFE35PWO74CPp3qAQ%253D%253D&n=1, Message: Method Not Allowed

I am able to reproduce the issue. Retrofit2 doesn’t accept ? in the @Path parameter and hence replaces it with %3F even though it’s not supposed to encode. This is because, ? is reserved for @Query params.

@kirangodishala is there a trailing-slash (or lack of trailing slash) issue around here in DockerRegistryNamedAccountCredentials?