emissary: Ambassador crashes when it fails to retrieve server APIs

Describe the bug The ambassador pod goes into crash-loop when it fails to connect to Kubernetes API server before eventually coming up.

+ kubewatch --namespace pavan --sync 'python3 /ambassador/post_update.py' --warmup-delay 10s secrets services
AMBASSADOR: diagd running
AMBASSADOR: waiting
PIDS: 30:ambex 31:diagd 49:kubewatch
2019/04/08 23:32:32 unable to retrieve the complete list of server APIs: custom.api.pavan/v1alpha1: the server is currently unable to handle the request
AMBASSADOR: kubewatch exited with status 1
AMBASSADOR: shutting down (1)

NAME                                  READY   STATUS    RESTARTS   AGE
gateway-7d9d88db4f-nhkqn              1/1     Running   4          49m

Ambassador deployment is named gateway in our setup.

To Reproduce Deploy the ambassador and the service responding to the custom Kubernetes APIs together. If the service is not yet up when ambassador starts watching the namespace, it crashes.

Expected behavior Ambassador should come up without crashing/restarts or there should be a way to configure it to not watch a particular service.

Versions

• Ambassador: 0.53.1
• Kubernetes environment: Google Kubernetes Engine
• Version: 1.12.5

Additional context The custom API service is designed to respond to kubectl requests and is not configured with any ambassador related annotations. Note that the same setup works fine with versions < 0.50+.