emissary: Ambadassor pod doesn't start due to failing readiness probe

Describe the bug On a completely naked K3S single-node cluster, it seems that the Ambadassor pod is not getting up because the configured readiness probe is failing. This is a new behaviour in version 1.10.0, using version 1.9.1. This also seems to be not related to the base operating system, I tried the same with Ubuntu 20.04 and experience the same issue.

...
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m29s                default-scheduler  Successfully assigned ambassador/ambassador-85d494566b-zfprw to k3os-6444
  Normal   Pulling    2m29s                kubelet            Pulling image "docker.io/datawire/aes:1.10.0"
  Normal   Pulled     107s                 kubelet            Successfully pulled image "docker.io/datawire/aes:1.10.0"
  Normal   Created    106s                 kubelet            Created container aes
  Normal   Started    106s                 kubelet            Started container aes
  Warning  Unhealthy  44s (x21 over 104s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503

Log output of that pod:

2021/01/07 14:37:08 Started Ambassador
2021/01/07 14:37:08 AMBASSADOR_CLUSTER_ID=82f6d5ed-63a3-5d00-a1a2-0094c451ffcd
time="2021-01-07 14:37:08" level=info msg="[pid:15] started command []string{\"diagd\", \"/ambassador/snapshots\", \"/ambassador/bootstrap-ads.json\", \"/ambassador/envoy/envoy.json\", \"--notices\", \"/ambassador/notices.json\", \"--port\", \"8004\", \"--kick\", \"kill -HUP 1\"}" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:143" CMD=entrypoint PID=1 THREAD=/diagd
time="2021-01-07 14:37:08" level=info msg="[pid:15] stdin  < not logging input read from file /dev/stdin" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:145" CMD=entrypoint PID=1 THREAD=/diagd
time="2021-01-07 14:37:08" level=info msg="[pid:15] stdout > not logging output written to file /dev/stdout" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:148" CMD=entrypoint PID=1 THREAD=/diagd
time="2021-01-07 14:37:08" level=info msg="[pid:15] stderr > not logging output written to file /dev/stderr" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:151" CMD=entrypoint PID=1 THREAD=/diagd
time="2021-01-07 14:37:08" level=info msg="[pid:16] started command []string{\"/ambassador/sidecars/amb-sidecar\"}" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:143" CMD=entrypoint PID=1 THREAD=/amb-sidecar
time="2021-01-07 14:37:08" level=info msg="[pid:16] stdin  < not logging input read from file /dev/stdin" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:145" CMD=entrypoint PID=1 THREAD=/amb-sidecar
time="2021-01-07 14:37:08" level=info msg="[pid:16] stdout > not logging output written to file /dev/stdout" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:148" CMD=entrypoint PID=1 THREAD=/amb-sidecar
time="2021-01-07 14:37:08" level=info msg="[pid:16] stderr > not logging output written to file /dev/stderr" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:151" CMD=entrypoint PID=1 THREAD=/amb-sidecar
time="2021-01-07 14:37:09" level=info msg="Ambassador Edge Stack configuration loaded" func=github.com/datawire/apro/cmd/amb-sidecar.runE file="github.com/datawire/apro/cmd/amb-sidecar/main.go:123" CMD=amb-sidecar PID=16
time="2021-01-07T14:37:10Z" level=warning msg="statsd is not in use"
time="2021-01-07 14:37:10" level=error msg="Failed to create watch on /home/ambassador/.config/ambassador/: Changes might require a restart: no such file or directory" func=github.com/datawire/apro/cmd/amb-sidecar.triggerOnChange file="github.com/datawire/apro/cmd/amb-sidecar/files.go:56" CMD=amb-sidecar PID=16 THREAD=/license_refresh
E0107 14:37:12.425241       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
time="2021-01-07 14:37:12" level=error msg="0 filters configured" func="github.com/datawire/apro/cmd/amb-sidecar/filters/controller.(*Controller).Watch.func1" file="github.com/datawire/apro/cmd/amb-sidecar/filters/controller/controller.go:150" CMD=amb-sidecar PID=16 THREAD=/auth_controller
2021/01/07 14:37:12 http: proxy error: dial tcp 127.0.0.1:8004: connect: connection refused
time="2021-01-07 14:37:12" level=error msg="Bad HTTP request: status_code=502" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1.1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:72" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
time="2021-01-07 14:37:12" level=error msg="HTTP error 502 from http://127.0.0.1:8877/ambassador/v0/diag/?json=true" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:79" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
2021-01-07 14:37:12 diagd 1.10.0 [P15TMainThread] INFO: AMBASSADOR_FAST_RECONFIGURE disabled, not initializing cache
2021-01-07 14:37:12 diagd 1.10.0 [P15TMainThread] INFO: WILL NOT update Mapping status
2021-01-07 14:37:12 diagd 1.10.0 [P15TMainThread] INFO: thread count 5, listening on 0.0.0.0:8004
E0107 14:37:13.473103       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0107 14:37:16.160640       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021-01-07 14:37:18 diagd 1.10.0 [P15TMainThread] INFO: Ambassador 1.10.0 booted
[2021-01-07 14:37:18 +0000] [15] [INFO] Starting gunicorn 20.0.4
[2021-01-07 14:37:18 +0000] [15] [INFO] Listening at: http://0.0.0.0:8004 (15)
[2021-01-07 14:37:18 +0000] [15] [INFO] Using worker: threads
[2021-01-07 14:37:18 +0000] [35] [INFO] Booting worker with pid: 35
2021-01-07 14:37:18 diagd 1.10.0 [P35TAEW] INFO: starting Scout checker and timer logger
2021-01-07 14:37:18 diagd 1.10.0 [P35TAEW] INFO: starting event watcher
2021/01/07 14:37:18 Memory Usage 0.15Gi (24%)
    PID 1, 0.06Gi: busyambassador entrypoint 
    PID 15, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1 
    PID 16, 0.11Gi: /ambassador/sidecars/amb-sidecar 
    PID 35, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1 
E0107 14:37:20.061177       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
Calling Metriton
E0107 14:37:27.762765       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0107 14:37:45.182925       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021-01-07 14:38:12 diagd 1.10.0 [P35TThreadPoolExecutor-0_0] ERROR: 07CBDB25-E63D-4E41-B9D5-13F502F76AE0: 127.0.0.1 "GET /ambassador/v0/diag/" 0ms 500 failure
time="2021-01-07 14:38:12" level=error msg="Bad HTTP request: status_code=500" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1.1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:72" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
time="2021-01-07 14:38:12" level=error msg="HTTP error 500 from http://127.0.0.1:8877/ambassador/v0/diag/?json=true" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:79" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
E0107 14:38:25.393432       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021-01-07 14:39:12 diagd 1.10.0 [P35TThreadPoolExecutor-0_0] ERROR: 33DC502C-F547-4127-9216-E43BC25A3F74: 127.0.0.1 "GET /ambassador/v0/diag/" 0ms 500 failure
time="2021-01-07 14:39:12" level=error msg="Bad HTTP request: status_code=500" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1.1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:72" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
time="2021-01-07 14:39:12" level=error msg="HTTP error 500 from http://127.0.0.1:8877/ambassador/v0/diag/?json=true" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:79" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
E0107 14:39:21.269592       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0107 14:40:00.064385       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021-01-07 14:40:12 diagd 1.10.0 [P35TThreadPoolExecutor-0_0] ERROR: ADA10DD0-8698-41AE-8C4E-BE0484EEE642: 127.0.0.1 "GET /ambassador/v0/diag/" 0ms 500 failure
time="2021-01-07 14:40:12" level=error msg="Bad HTTP request: status_code=500" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1.1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:72" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
time="2021-01-07 14:40:12" level=error msg="HTTP error 500 from http://127.0.0.1:8877/ambassador/v0/diag/?json=true" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:79" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
E0107 14:40:38.978127       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021-01-07 14:41:12 diagd 1.10.0 [P35TThreadPoolExecutor-0_0] ERROR: F957E7CC-8C70-4C26-83B6-AECF3B02EA27: 127.0.0.1 "GET /ambassador/v0/diag/" 0ms 500 failure
time="2021-01-07 14:41:12" level=error msg="Bad HTTP request: status_code=500" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1.1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:72" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
time="2021-01-07 14:41:12" level=error msg="HTTP error 500 from http://127.0.0.1:8877/ambassador/v0/diag/?json=true" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:79" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
E0107 14:41:31.556394       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0107 14:42:07.755089       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021-01-07 14:42:12 diagd 1.10.0 [P35TThreadPoolExecutor-0_0] ERROR: 40C09CC7-37F9-43FF-AB6A-245F985E5079: 127.0.0.1 "GET /ambassador/v0/diag/" 0ms 500 failure
time="2021-01-07 14:42:12" level=error msg="Bad HTTP request: status_code=500" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1.1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:72" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
time="2021-01-07 14:42:12" level=error msg="HTTP error 500 from http://127.0.0.1:8877/ambassador/v0/diag/?json=true" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:79" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
E0107 14:43:03.716303       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021-01-07 14:43:12 diagd 1.10.0 [P35TThreadPoolExecutor-0_0] ERROR: 6D6AC540-0FA4-4DDA-8904-DB7556DA77BB: 127.0.0.1 "GET /ambassador/v0/diag/" 0ms 500 failure
time="2021-01-07 14:43:12" level=error msg="Bad HTTP request: status_code=500" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1.1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:72" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
time="2021-01-07 14:43:12" level=error msg="HTTP error 500 from http://127.0.0.1:8877/ambassador/v0/diag/?json=true" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:79" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
E0107 14:43:38.468523       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0107 14:44:10.853457       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021-01-07 14:44:12 diagd 1.10.0 [P35TThreadPoolExecutor-0_0] ERROR: 1B445D22-97F4-42E6-9C6A-99486B66A4F5: 127.0.0.1 "GET /ambassador/v0/diag/" 0ms 500 failure
time="2021-01-07 14:44:12" level=error msg="Bad HTTP request: status_code=500" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1.1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:72" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
time="2021-01-07 14:44:12" level=error msg="HTTP error 500 from http://127.0.0.1:8877/ambassador/v0/diag/?json=true" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:79" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
E0107 14:44:49.900943       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021-01-07 14:45:12 diagd 1.10.0 [P35TThreadPoolExecutor-0_0] ERROR: 690AE2C7-FDE4-4C64-B9BA-98434A8F5F37: 127.0.0.1 "GET /ambassador/v0/diag/" 0ms 500 failure
time="2021-01-07 14:45:12" level=error msg="Bad HTTP request: status_code=500" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1.1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:72" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
time="2021-01-07 14:45:12" level=error msg="HTTP error 500 from http://127.0.0.1:8877/ambassador/v0/diag/?json=true" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:79" CMD=amb-sidecar PID=16 component=devportal mhost= url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true"
E0107 14:45:32.596711       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

To Reproduce Install Ambassador (YAML method) on k3sos (I used v0.11.1)

Expected behavior Readiness probe is successful.

Versions (please complete the following information):

• Ambassador: 1.10.0
• Kubernetes environment: bare metal, k3os VM
• Kubernetes version: v1.18.9+k3s1
• Kubernetes OS-Image: k3OS v0.11.1
• Kubernetes Kernel-Version: 5.4.0-48-generic
• Kubernetes Container-Runtime: containerd://1.3.3-k3s2