spinnaker: Pipeline execution requests via webhook triggers are lost

Issue Summary:

Aloha!

We are seeing an issue where pipeline execution requests via webhook triggers are lost. After making an HTTP request to the configured webhook URL, we do see a successful 200 status code, with a response that looks like:

{"eventProcessed":true,"eventId":"9c43e13a-df5b-41a5-aca6-cf43f9a36416"}

However, the pipeline is actually never executed (we do not see anything in the application’s web page, or via spin pipeline execution list using the spin CLI). Retrying after a minute or so (with the exact same request payload) does seems to work.

This happens almost every single time when creating a brand new pipeline, and then immediately proceeding to run it (i.e., as part of an application bootstrap script that installs a set of pipelines, then runs one of them). This also occasionally happens for pipelines that were created a long time ago.

Note that when this happens, we do not see any interesting logs for orca or elsewhere pertaining to the returned eventId or the chosen pipeline’s name or ID.

Cloud Provider(s):

GCP

Environment:

Spinnaker deployed into GKE cluster.

Feature Area:

Pipelines

Steps to Reproduce:

• Create a new pipeline with a webhook trigger
• Immediately call that webhook trigger

Additional Details:

We are seeing this across multiple Spinnaker versions, including 1.18.2, 1.19.6, 1.20.4.