ingress-nginx: Premature readiness probe success due to race condition in check for backends initialization, causing 404s

NGINX Ingress controller version

bash-5.1$ /nginx-ingress-controller --version  
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.1.2
  Build:         bab0fbab0c1a7c3641bd379f27857113d574d904
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.9

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version): 1.21

Environment:

• Cloud provider or hardware configuration: Bare metal (on-premises)
• OS (e.g. from /etc/os-release): Debian 10 (buster)
• Kernel (e.g. uname -a): 5.10.0-13-amd64
• Install tools: N/A
• Basic cluster related info: N/A
• How was the ingress-nginx-controller installed: Manifests. There are multiple instances of ingress-nginx using different ingressClassName and IngressClass .spec.controller.
• Current state of ingress object, if applicable: There are thousands of them, this is probably triggering the issue 😃

What happened:

One of our ingress classes has ~3k associated ingress objects. When a new ingress pod for this class starts up, it returns 404s for backends for a brief period of time, even after passing the readiness probe. We have increased the readinessProbe initialDelaySeconds to 40, which helps, but feels like a band-aid.

What you expected to happen:

The readiness probe should not pass until the upstreams are fully synchronized.

How to reproduce it:

I am working on a reproducer, but i think the actual issue is here:

1. The readiness probe is checking that the backends data is set via configuration.get_backends_data().
2. When the backends are POSTed by the controller, this variable is set directly but there is actually an asynchronous syncronization loop that later applies these backends to the underlying nginx proxy upstreams.
3. This sync runs every second. But with 3000+ ingresses, many with multiple hosts (multiple server blocks in resulting nginx config), i am not actually sure how long a single sync takes (i guess it could be many seconds?).
4. During the gap between these two, the pod is reporting ready but is serving 404s. This adds the pod to the service endpoints, and advertises it with BGP in our datacenter. Clients get 404s 😭.