spinnaker: Igor threshold blocking GCR triggers for repos w/ > 1000 images
igor threshold blocking triggers for GCRs with more than 1000 tags. assuming related to:
- https://github.com/spinnaker/igor/commit/5ae4fff
- https://github.com/spinnaker/igor/blob/master/igor-web/src/main/groovy/com/netflix/spinnaker/igor/IgorConfigurationProperties.groovy#L47
guessing that all tags showing up as new tags at first after the upgrade is not desired - eg threshold working as expected and protecting us from possible issues like the flurry of triggers noted in issue 2? i have not restarted services to see if it is a one time thing, or will happen each clouddriver/igor? restart.
example under threshold:
$ gcloud logging read --project spinnaker-gcp-project 'resource.type="container" resource.labels.cluster_name="spinnaker" "new images for project-a-gcr" timestamp>="2018-05-08T18:30:00Z"' --format 'value(textPayload)' --limit 5 --order asc
2018-05-08 18:45:46.357 INFO 1 --- [readScheduler-2] c.n.spinnaker.igor.docker.DockerMonitor : Found 254 new images for project-a-gcr
2018-05-08 18:46:21.483 INFO 1 --- [readScheduler-2] c.n.spinnaker.igor.docker.DockerMonitor : Found 0 new images for project-a-gcr
2018-05-08 18:46:42.828 INFO 1 --- [readScheduler-2] c.n.spinnaker.igor.docker.DockerMonitor : Found 0 new images for project-a-gcr
2018-05-08 18:47:13.446 INFO 1 --- [readScheduler-2] c.n.spinnaker.igor.docker.DockerMonitor : Found 0 new images for project-a-gcr
2018-05-08 18:47:43.466 INFO 1 --- [readScheduler-2] c.n.spinnaker.igor.docker.DockerMonitor : Found 0 new images for project-a-gcr
example over threshold:
$ gcloud logging read --project spinnaker-gcp-project 'resource.type="container" resource.labels.cluster_name="spinnaker" "new images for project-b-gcr" timestamp>="2018-05-08T18:30:00Z"' --format 'value(textPayload)' --limit 5 --order asc
2018-05-08 18:45:52.661 INFO 1 --- [readScheduler-2] c.n.spinnaker.igor.docker.DockerMonitor : Found 10678 new images for project-b-gcr
2018-05-08 18:46:27.557 INFO 1 --- [readScheduler-2] c.n.spinnaker.igor.docker.DockerMonitor : Found 10678 new images for project-b-gcr
2018-05-08 18:46:46.389 INFO 1 --- [readScheduler-2] c.n.spinnaker.igor.docker.DockerMonitor : Found 10678 new images for project-b-gcr
2018-05-08 18:47:16.845 INFO 1 --- [readScheduler-2] c.n.spinnaker.igor.docker.DockerMonitor : Found 10678 new images for project-b-gcr
2018-05-08 18:47:46.261 INFO 1 --- [readScheduler-2] c.n.spinnaker.igor.docker.DockerMonitor : Found 10678 new images for project-b-gcr
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 6
- Comments: 15 (3 by maintainers)
This issue is tagged as ‘to-be-closed’ and hasn’t been updated in 45 days, so we are closing it. You can always reopen this issue if needed.
For now, this can be manually fixed by “fast forwarding” the new accounts. This means all events from a docker account are accepted w/o sending any notifications to echo for pipeline triggers.
This is done by sending
as shown here.
Actually, after upgrading to Spinnaker 1.7.*, our pipeline triggers stopped working, as well (1.6.1 works fine). Our images are pushed to a GCR repo with far more than 1000 tags and the trigger is set up for images in this repo. I can still see all the tags when manually starting a pipeline, but the triggers do not work. Is there any way I can find out if this is related to this issue (I actually thought that we do not use any igor/echo functionality) and will your workaround thus work for us? Thanks for the help.
This issue is tagged as ‘stale’ and hasn’t been updated in 53 days, so we are tagging it as ‘to-be-closed’. It will be closed in 45 days unless updates are made. If you want to remove this label, comment:
This issue hasn’t been updated in 45 days, so we are tagging it as ‘stale’. If you want to remove this label, comment:
@rossedman I use Halyard to deploy Spinnaker to GKE. How would I tell it to deploy two clouddriver POD replicas?
(Aside from just modifying the RS spec myself after Halyard deploys it)