spinnaker: Clouddriver: stops syncing Docker registry at intermittent intervals

Issue Summary:

Potentially related issue: #4780

Clouddriver will stop syncing with our Docker registry at intermittent intervals. There is no error message logged by Clouddriver, event after setting the log level to TRACE.

Cloud Provider(s):

AWS EKS

Environment:

• EKS v1.18
• Halyard 1.40.0
• Spinnaker 1.23.3
• Clouddriver 7.0.3

Feature Area:

Clouddriver

Description:

Steps to Reproduce:

Additional Details:

When working correctly, Clouddriver will output a info message stating that it is going to start describing the cached items in the registry, and follow up with the amount of ids and tags it grabs:

2020-12-14 07:01:10.838  INFO 1 --- [cutionAction-29] .d.r.p.a.DockerRegistryImageCachingAgent : Describing items in <docker-registry>/DockerRegistryImageCachingAgent[1/1]
2020-12-14 07:01:13.103  INFO 1 --- [cutionAction-29] .d.r.p.a.DockerRegistryImageCachingAgent : Caching 21174 tagged images in <docker-registry>/DockerRegistryImageCachingAgent[1/1]
2020-12-14 07:01:13.103  INFO 1 --- [cutionAction-29] .d.r.p.a.DockerRegistryImageCachingAgent : Caching 21174 image ids in <docker-registry>/DockerRegistryImageCachingAgent[1/1]

However, I’ve noticed that after Clouddriver stops syncing the registry, the cached tags and ids are not printing, and we only get the message from the describe call:

2020-12-13 08:09:57.750  INFO 1 --- [utionAction-116] .d.r.p.a.DockerRegistryImageCachingAgent : Describing items in <docker-registry>/DockerRegistryImageCachingAgent[1/1]

From this point onwards, Clouddriver will not sync with the registry until it (Clouddriver) has been restarted. Once it has been restarted, it will sync with the registry just fine until this intermittent failure occurs again.

It seems like if there is a problem here with Clouddriver proper, the offending code would be somewhere in the buildCacheResult method of the DockerRegistryImageCachingAgent: https://github.com/spinnaker/clouddriver/blob/874cef3644c826a1932d9670d59ccb77a19e19a2/clouddriver-docker/src/main/groovy/com/netflix/spinnaker/clouddriver/docker/registry/provider/agent/DockerRegistryImageCachingAgent.groovy#L135

I have also checked connectivity to our Docker registry during these outages, and I can connect to it just fine while this issue occurs with Clouddriver.