promo-tools: image promotion is very slow despite low resource utilization

When running in Kubernetes CI image promotion takes more than one hour, often multiple hours.

Worse, it takes an hour just running presubmit tests in k8s.io repo. As a result a quickly approved image promotion PR may take 3 hours or more to take affect.

I took a cursory pass at this by: https://github.com/kubernetes/test-infra/pull/27743 + https://github.com/kubernetes/test-infra/pull/27765 which set large (7 core, 40 Gi RAM) allocations, configure GOMAXPROCS to match, and increase --threads (worker goroutines).

Since those changes, we can follow from a running job, to it’s pod, to the node the pod is on, to the backing GCE VM. On that VM, there should only be kubelet / system daemons, other workloads will not be scheduled here due to essentially no schedule-able resources being available (VM has 8 cores, we request 7, some are reserved for system agents).

We can see that CPU usage spikes when the job starts (git cloning?) and then the entire machine settles to just 3-7% utilization, outgoing connections only ~2/s to ~3/s, disk write < 0.6MiB/s, network egress < 0.7MiB/s far below the initial spike:

This implies we are bottlenecking heavily on something like waiting for the sigstore API …?

Copying images could otherwise conceivably bottleneck on CPU or Network but both of those are hardly in use.

I half-joked about synchronized logging and the volume of logs, but I don’t seriously think it’s contention over the logger.

Besides investigating what exactly is causing this slow-down, I think we should consider enabling running N postsubmit jobs each responsible for a subset of the images, only triggered when the respective manifest directories change, based on run_if_changed plus some option to the image promoter to specify which manifests to read.