amazon-vpc-cni-k8s: CNI fails healthchecks, errors when we run containers

Hello, we are experiencing errors in aws cni, in particular from logs I can point out two possible major issues:

1. Excessive amount of lines about “ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary”. It’s just does not look right for me, but I could not find an answer or figure out what could be an issue of it.
2. Another is when in logs appeared lots of messages about DelNetwork related to cron container. These lines also do not look good for me, because exactly when they appear, aws-cni shows failed probe.

==> /var/log/aws-routed-eni/ipamd.log <==
{"level":"debug","ts":"2021-08-25T15:58:52.082Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T15:58:57.082Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T15:59:03.744Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T15:59:08.764Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T15:59:13.765Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T15:59:18.765Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T15:59:23.765Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T15:59:28.766Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T15:59:34.454Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T15:59:39.508Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T15:59:44.601Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T15:59:49.602Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T15:59:52.518Z","caller":"ipamd/ipamd.go:557","msg":"Reconciling ENI/IP pool info because time since last 1m2.942069324s <= 1m0s"}
{"level":"debug","ts":"2021-08-25T15:59:53.840Z","caller":"ipamd/ipamd.go:1106","msg":"Total number of interfaces found: 1 "}
{"level":"debug","ts":"2021-08-25T15:59:53.840Z","caller":"awsutils/awsutils.go:539","msg":"Found ENI MAC address: 02:72:23:ac:41:1b"}
{"level":"debug","ts":"2021-08-25T15:59:53.842Z","caller":"awsutils/awsutils.go:539","msg":"Found ENI: eni-0488a0403d6fa7e41, MAC 02:72:23:ac:41:1b, device 0"}
{"level":"debug","ts":"2021-08-25T15:59:53.844Z","caller":"ipamd/ipamd.go:557","msg":"Reconcile existing ENI eni-0488a0403d6fa7e41 IP pool"}
{"level":"info","ts":"2021-08-25T15:59:53.844Z","caller":"ipamd/ipamd.go:1239","msg":"Reconcile and skip primary IP 172.16.2.235 on ENI eni-0488a0403d6fa7e41"}
{"level":"debug","ts":"2021-08-25T15:59:53.844Z","caller":"ipamd/ipamd.go:557","msg":"Reconcile existing ENI eni-0488a0403d6fa7e41 IP prefixes"}
{"level":"debug","ts":"2021-08-25T15:59:53.844Z","caller":"ipamd/ipamd.go:1172","msg":"Found prefix pool count 3 for eni eni-0488a0403d6fa7e41\n"}
{"level":"debug","ts":"2021-08-25T15:59:53.844Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.160/28"}
{"level":"info","ts":"2021-08-25T15:59:53.844Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.160/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T15:59:53.844Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T15:59:53.844Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.16/28"}
{"level":"info","ts":"2021-08-25T15:59:53.844Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.16/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T15:59:53.844Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T15:59:53.844Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.176/28"}
{"level":"info","ts":"2021-08-25T15:59:53.845Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.176/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T15:59:53.845Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T15:59:53.845Z","caller":"ipamd/ipamd.go:557","msg":"Successfully Reconciled ENI/IP pool"}
{"level":"debug","ts":"2021-08-25T15:59:53.845Z","caller":"ipamd/ipamd.go:557","msg":"IP/Prefix Address Pool stats: total: 48, assigned: 15, total prefixes: 3"}
{"level":"debug","ts":"2021-08-25T15:59:56.345Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T15:59:56.509Z","caller":"awsutils/awsutils.go:1480","msg":"Checking for leaked AWS CNI ENIs."}
{"level":"debug","ts":"2021-08-25T15:59:56.738Z","caller":"ec2/api.go:21868","msg":"EC2 DescribeNetworkInterfaces succeeded with 0 results on page 1"}
{"level":"debug","ts":"2021-08-25T15:59:56.738Z","caller":"awsutils/awsutils.go:1489","msg":"No AWS CNI leaked ENIs found."}
{"level":"debug","ts":"2021-08-25T16:00:01.350Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:00:06.350Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:00:11.351Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:00:16.351Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:00:21.351Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:00:26.352Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:00:31.352Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:00:36.353Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:00:41.353Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:00:46.353Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:00:51.354Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:00:53.854Z","caller":"ipamd/ipamd.go:557","msg":"Reconciling ENI/IP pool info because time since last 1m1.336181777s <= 1m0s"}
{"level":"debug","ts":"2021-08-25T16:00:53.855Z","caller":"ipamd/ipamd.go:1106","msg":"Total number of interfaces found: 1 "}
{"level":"debug","ts":"2021-08-25T16:00:53.855Z","caller":"awsutils/awsutils.go:539","msg":"Found ENI MAC address: 02:72:23:ac:41:1b"}
{"level":"debug","ts":"2021-08-25T16:00:53.858Z","caller":"awsutils/awsutils.go:539","msg":"Found ENI: eni-0488a0403d6fa7e41, MAC 02:72:23:ac:41:1b, device 0"}
{"level":"debug","ts":"2021-08-25T16:00:53.863Z","caller":"ipamd/ipamd.go:557","msg":"Reconcile existing ENI eni-0488a0403d6fa7e41 IP pool"}
{"level":"info","ts":"2021-08-25T16:00:53.864Z","caller":"ipamd/ipamd.go:1239","msg":"Reconcile and skip primary IP 172.16.2.235 on ENI eni-0488a0403d6fa7e41"}
{"level":"debug","ts":"2021-08-25T16:00:53.864Z","caller":"ipamd/ipamd.go:557","msg":"Reconcile existing ENI eni-0488a0403d6fa7e41 IP prefixes"}
{"level":"debug","ts":"2021-08-25T16:00:53.864Z","caller":"ipamd/ipamd.go:1172","msg":"Found prefix pool count 3 for eni eni-0488a0403d6fa7e41\n"}
{"level":"debug","ts":"2021-08-25T16:00:53.864Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.160/28"}
{"level":"info","ts":"2021-08-25T16:00:53.864Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.160/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:00:53.864Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:00:53.864Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.16/28"}
{"level":"info","ts":"2021-08-25T16:00:53.864Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.16/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:00:53.864Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:00:53.864Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.176/28"}
{"level":"info","ts":"2021-08-25T16:00:53.864Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.176/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:00:53.864Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:00:53.864Z","caller":"ipamd/ipamd.go:557","msg":"Successfully Reconciled ENI/IP pool"}
{"level":"debug","ts":"2021-08-25T16:00:53.864Z","caller":"ipamd/ipamd.go:557","msg":"IP/Prefix Address Pool stats: total: 48, assigned: 15, total prefixes: 3"}
{"level":"debug","ts":"2021-08-25T16:00:56.364Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:01:01.364Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:01:06.365Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:01:11.365Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:01:16.365Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:01:21.366Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:01:26.366Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:01:31.366Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:01:36.367Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:01:41.367Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:01:46.367Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:01:51.368Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:01:53.868Z","caller":"ipamd/ipamd.go:557","msg":"Reconciling ENI/IP pool info because time since last 1m0.014229104s <= 1m0s"}
{"level":"debug","ts":"2021-08-25T16:01:53.869Z","caller":"ipamd/ipamd.go:1106","msg":"Total number of interfaces found: 1 "}
{"level":"debug","ts":"2021-08-25T16:01:53.869Z","caller":"awsutils/awsutils.go:539","msg":"Found ENI MAC address: 02:72:23:ac:41:1b"}
{"level":"debug","ts":"2021-08-25T16:01:53.872Z","caller":"awsutils/awsutils.go:539","msg":"Found ENI: eni-0488a0403d6fa7e41, MAC 02:72:23:ac:41:1b, device 0"}
{"level":"debug","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:557","msg":"Reconcile existing ENI eni-0488a0403d6fa7e41 IP pool"}
{"level":"info","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:1239","msg":"Reconcile and skip primary IP 172.16.2.235 on ENI eni-0488a0403d6fa7e41"}
{"level":"debug","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:557","msg":"Reconcile existing ENI eni-0488a0403d6fa7e41 IP prefixes"}
{"level":"debug","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:1172","msg":"Found prefix pool count 3 for eni eni-0488a0403d6fa7e41\n"}
{"level":"debug","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.160/28"}
{"level":"info","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.160/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.16/28"}
{"level":"info","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.16/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.176/28"}
{"level":"info","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.176/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:557","msg":"Successfully Reconciled ENI/IP pool"}
{"level":"debug","ts":"2021-08-25T16:01:53.874Z","caller":"ipamd/ipamd.go:557","msg":"IP/Prefix Address Pool stats: total: 48, assigned: 15, total prefixes: 3"}
{"level":"debug","ts":"2021-08-25T16:01:56.374Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:02:01.375Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:02:06.375Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:02:11.376Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:02:16.376Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:02:21.376Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:02:26.377Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:02:31.377Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:02:36.378Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:02:41.378Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:02:46.378Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:02:51.379Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:02:53.879Z","caller":"ipamd/ipamd.go:557","msg":"Reconciling ENI/IP pool info because time since last 1m0.010720268s <= 1m0s"}
{"level":"debug","ts":"2021-08-25T16:02:53.880Z","caller":"ipamd/ipamd.go:1106","msg":"Total number of interfaces found: 1 "}
{"level":"debug","ts":"2021-08-25T16:02:53.880Z","caller":"awsutils/awsutils.go:539","msg":"Found ENI MAC address: 02:72:23:ac:41:1b"}
{"level":"debug","ts":"2021-08-25T16:02:53.882Z","caller":"awsutils/awsutils.go:539","msg":"Found ENI: eni-0488a0403d6fa7e41, MAC 02:72:23:ac:41:1b, device 0"}
{"level":"debug","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:557","msg":"Reconcile existing ENI eni-0488a0403d6fa7e41 IP pool"}
{"level":"info","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:1239","msg":"Reconcile and skip primary IP 172.16.2.235 on ENI eni-0488a0403d6fa7e41"}
{"level":"debug","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:557","msg":"Reconcile existing ENI eni-0488a0403d6fa7e41 IP prefixes"}
{"level":"debug","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:1172","msg":"Found prefix pool count 3 for eni eni-0488a0403d6fa7e41\n"}
{"level":"debug","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.160/28"}
{"level":"info","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.160/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.16/28"}
{"level":"info","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.16/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.176/28"}
{"level":"info","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.176/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:557","msg":"Successfully Reconciled ENI/IP pool"}
{"level":"debug","ts":"2021-08-25T16:02:53.884Z","caller":"ipamd/ipamd.go:557","msg":"IP/Prefix Address Pool stats: total: 48, assigned: 15, total prefixes: 3"}
{"level":"debug","ts":"2021-08-25T16:02:56.384Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:03:01.384Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:03:06.385Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:03:11.385Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:03:16.385Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:03:21.386Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:03:26.386Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:03:31.386Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:03:36.387Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:03:41.387Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:03:46.388Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}

==> /var/log/aws-routed-eni/plugin.log <==
{"level":"info","ts":"2021-08-25T16:03:49.556Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(881db247ff8912d789852925501854632162de4d144cf6534dc12a76ce0f0c8a) Netns(/proc/30392/ns/net) IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-03-cron-27165117-mrcpg;K8S_POD_INFRA_CONTAINER_ID=881db247ff8912d789852925501854632162de4d144cf6534dc12a76ce0f0c8a) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}

==> /var/log/aws-routed-eni/ipamd.log <==
{"level":"info","ts":"2021-08-25T16:03:49.565Z","caller":"rpc/rpc.pb.go:519","msg":"Received DelNetwork for Sandbox 881db247ff8912d789852925501854632162de4d144cf6534dc12a76ce0f0c8a"}
{"level":"debug","ts":"2021-08-25T16:03:49.565Z","caller":"rpc/rpc.pb.go:519","msg":"DelNetworkRequest: ClientVersion:\"v1.9.0\" K8S_POD_NAME:\"md-conqueror-satellite-pl-03-cron-27165117-mrcpg\" K8S_POD_NAMESPACE:\"staging-md-conqueror-satellite-3674\" K8S_POD_INFRA_CONTAINER_ID:\"881db247ff8912d789852925501854632162de4d144cf6534dc12a76ce0f0c8a\" Reason:\"PodDeleted\" ContainerID:\"881db247ff8912d789852925501854632162de4d144cf6534dc12a76ce0f0c8a\" IfName:\"eth0\" NetworkName:\"aws-cni\" "}
{"level":"debug","ts":"2021-08-25T16:03:49.565Z","caller":"ipamd/rpc_handler.go:204","msg":"UnassignPodIPv4Address: IP address pool stats: total:48, assigned 15, sandbox aws-cni/881db247ff8912d789852925501854632162de4d144cf6534dc12a76ce0f0c8a/eth0"}
{"level":"info","ts":"2021-08-25T16:03:49.565Z","caller":"datastore/data_store.go:982","msg":"UnAssignPodIPv4Address: Unassign IP 172.16.2.160 from sandbox aws-cni/881db247ff8912d789852925501854632162de4d144cf6534dc12a76ce0f0c8a/eth0"}
{"level":"info","ts":"2021-08-25T16:03:49.566Z","caller":"ipamd/rpc_handler.go:204","msg":"UnassignPodIPv4Address: sandbox aws-cni/881db247ff8912d789852925501854632162de4d144cf6534dc12a76ce0f0c8a/eth0's ipAddr 172.16.2.160, DeviceNumber 0"}
{"level":"info","ts":"2021-08-25T16:03:49.566Z","caller":"rpc/rpc.pb.go:519","msg":"Send DelNetworkReply: IPv4Addr 172.16.2.160, DeviceNumber: 0, err: <nil>"}

==> /var/log/aws-routed-eni/plugin.log <==
{"level":"info","ts":"2021-08-25T16:03:49.603Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received del network response for pod md-conqueror-satellite-pl-03-cron-27165117-mrcpg namespace staging-md-conqueror-satellite-3674 sandbox 881db247ff8912d789852925501854632162de4d144cf6534dc12a76ce0f0c8a: Success:true IPv4Addr:\"172.16.2.160\" "}
{"level":"debug","ts":"2021-08-25T16:03:49.606Z","caller":"routed-eni-cni-plugin/cni.go:316","msg":"TeardownNS: addr 172.16.2.160/32, deviceNumber 0"}
{"level":"info","ts":"2021-08-25T16:03:49.607Z","caller":"driver/driver.go:421","msg":"Delete toContainer rule for 172.16.2.160/32 "}
{"level":"debug","ts":"2021-08-25T16:03:49.607Z","caller":"driver/driver.go:421","msg":"Tear down of NS complete"}
{"level":"info","ts":"2021-08-25T16:03:49.631Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(64a8f3035030cfaefb3c8e92bbafca4b3532f60122972026b781968fca64ceec) Netns(/proc/30345/ns/net) IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165117-9wsfv;K8S_POD_INFRA_CONTAINER_ID=64a8f3035030cfaefb3c8e92bbafca4b3532f60122972026b781968fca64ceec) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}

==> /var/log/aws-routed-eni/ipamd.log <==
{"level":"info","ts":"2021-08-25T16:03:49.633Z","caller":"rpc/rpc.pb.go:519","msg":"Received DelNetwork for Sandbox 64a8f3035030cfaefb3c8e92bbafca4b3532f60122972026b781968fca64ceec"}
{"level":"debug","ts":"2021-08-25T16:03:49.633Z","caller":"rpc/rpc.pb.go:519","msg":"DelNetworkRequest: ClientVersion:\"v1.9.0\" K8S_POD_NAME:\"md-conqueror-satellite-de-04-cron-27165117-9wsfv\" K8S_POD_NAMESPACE:\"staging-md-conqueror-satellite-3674\" K8S_POD_INFRA_CONTAINER_ID:\"64a8f3035030cfaefb3c8e92bbafca4b3532f60122972026b781968fca64ceec\" Reason:\"PodDeleted\" ContainerID:\"64a8f3035030cfaefb3c8e92bbafca4b3532f60122972026b781968fca64ceec\" IfName:\"eth0\" NetworkName:\"aws-cni\" "}
{"level":"debug","ts":"2021-08-25T16:03:49.633Z","caller":"ipamd/rpc_handler.go:204","msg":"UnassignPodIPv4Address: IP address pool stats: total:48, assigned 14, sandbox aws-cni/64a8f3035030cfaefb3c8e92bbafca4b3532f60122972026b781968fca64ceec/eth0"}
{"level":"info","ts":"2021-08-25T16:03:49.633Z","caller":"datastore/data_store.go:982","msg":"UnAssignPodIPv4Address: Unassign IP 172.16.2.165 from sandbox aws-cni/64a8f3035030cfaefb3c8e92bbafca4b3532f60122972026b781968fca64ceec/eth0"}
{"level":"info","ts":"2021-08-25T16:03:49.634Z","caller":"ipamd/rpc_handler.go:204","msg":"UnassignPodIPv4Address: sandbox aws-cni/64a8f3035030cfaefb3c8e92bbafca4b3532f60122972026b781968fca64ceec/eth0's ipAddr 172.16.2.165, DeviceNumber 0"}
{"level":"info","ts":"2021-08-25T16:03:49.634Z","caller":"rpc/rpc.pb.go:519","msg":"Send DelNetworkReply: IPv4Addr 172.16.2.165, DeviceNumber: 0, err: <nil>"}

==> /var/log/aws-routed-eni/plugin.log <==
{"level":"info","ts":"2021-08-25T16:03:49.635Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received del network response for pod md-conqueror-satellite-de-04-cron-27165117-9wsfv namespace staging-md-conqueror-satellite-3674 sandbox 64a8f3035030cfaefb3c8e92bbafca4b3532f60122972026b781968fca64ceec: Success:true IPv4Addr:\"172.16.2.165\" "}
{"level":"debug","ts":"2021-08-25T16:03:49.635Z","caller":"routed-eni-cni-plugin/cni.go:316","msg":"TeardownNS: addr 172.16.2.165/32, deviceNumber 0"}
{"level":"info","ts":"2021-08-25T16:03:49.635Z","caller":"driver/driver.go:421","msg":"Delete toContainer rule for 172.16.2.165/32 "}
{"level":"debug","ts":"2021-08-25T16:03:49.635Z","caller":"driver/driver.go:421","msg":"Tear down of NS complete"}

==> /var/log/aws-routed-eni/ipamd.log <==
{"level":"debug","ts":"2021-08-25T16:03:51.388Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}

==> /var/log/aws-routed-eni/plugin.log <==
{"level":"info","ts":"2021-08-25T16:03:51.810Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(0f8601ce311e1c90e4417baaded96f7ef6c4b4ff933a947f73b74d71a8399fb0) Netns(/proc/30363/ns/net) IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-01-cron-27165117-qhf47;K8S_POD_INFRA_CONTAINER_ID=0f8601ce311e1c90e4417baaded96f7ef6c4b4ff933a947f73b74d71a8399fb0) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}

==> /var/log/aws-routed-eni/ipamd.log <==
{"level":"info","ts":"2021-08-25T16:03:51.813Z","caller":"rpc/rpc.pb.go:519","msg":"Received DelNetwork for Sandbox 0f8601ce311e1c90e4417baaded96f7ef6c4b4ff933a947f73b74d71a8399fb0"}
{"level":"debug","ts":"2021-08-25T16:03:51.813Z","caller":"rpc/rpc.pb.go:519","msg":"DelNetworkRequest: ClientVersion:\"v1.9.0\" K8S_POD_NAME:\"md-conqueror-satellite-pl-01-cron-27165117-qhf47\" K8S_POD_NAMESPACE:\"staging-md-conqueror-satellite-3674\" K8S_POD_INFRA_CONTAINER_ID:\"0f8601ce311e1c90e4417baaded96f7ef6c4b4ff933a947f73b74d71a8399fb0\" Reason:\"PodDeleted\" ContainerID:\"0f8601ce311e1c90e4417baaded96f7ef6c4b4ff933a947f73b74d71a8399fb0\" IfName:\"eth0\" NetworkName:\"aws-cni\" "}
{"level":"debug","ts":"2021-08-25T16:03:51.813Z","caller":"ipamd/rpc_handler.go:204","msg":"UnassignPodIPv4Address: IP address pool stats: total:48, assigned 13, sandbox aws-cni/0f8601ce311e1c90e4417baaded96f7ef6c4b4ff933a947f73b74d71a8399fb0/eth0"}
{"level":"info","ts":"2021-08-25T16:03:51.813Z","caller":"datastore/data_store.go:982","msg":"UnAssignPodIPv4Address: Unassign IP 172.16.2.161 from sandbox aws-cni/0f8601ce311e1c90e4417baaded96f7ef6c4b4ff933a947f73b74d71a8399fb0/eth0"}
{"level":"info","ts":"2021-08-25T16:03:51.813Z","caller":"ipamd/rpc_handler.go:204","msg":"UnassignPodIPv4Address: sandbox aws-cni/0f8601ce311e1c90e4417baaded96f7ef6c4b4ff933a947f73b74d71a8399fb0/eth0's ipAddr 172.16.2.161, DeviceNumber 0"}
{"level":"info","ts":"2021-08-25T16:03:51.813Z","caller":"rpc/rpc.pb.go:519","msg":"Send DelNetworkReply: IPv4Addr 172.16.2.161, DeviceNumber: 0, err: <nil>"}

==> /var/log/aws-routed-eni/plugin.log <==
{"level":"info","ts":"2021-08-25T16:03:51.814Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received del network response for pod md-conqueror-satellite-pl-01-cron-27165117-qhf47 namespace staging-md-conqueror-satellite-3674 sandbox 0f8601ce311e1c90e4417baaded96f7ef6c4b4ff933a947f73b74d71a8399fb0: Success:true IPv4Addr:\"172.16.2.161\" "}
{"level":"debug","ts":"2021-08-25T16:03:51.814Z","caller":"routed-eni-cni-plugin/cni.go:316","msg":"TeardownNS: addr 172.16.2.161/32, deviceNumber 0"}
{"level":"info","ts":"2021-08-25T16:03:51.814Z","caller":"driver/driver.go:421","msg":"Delete toContainer rule for 172.16.2.161/32 "}
{"level":"debug","ts":"2021-08-25T16:03:51.814Z","caller":"driver/driver.go:421","msg":"Tear down of NS complete"}

==> /var/log/aws-routed-eni/ipamd.log <==
{"level":"debug","ts":"2021-08-25T16:03:53.888Z","caller":"ipamd/ipamd.go:557","msg":"Reconciling ENI/IP pool info because time since last 1m0.009301766s <= 1m0s"}
{"level":"debug","ts":"2021-08-25T16:03:53.889Z","caller":"ipamd/ipamd.go:1106","msg":"Total number of interfaces found: 1 "}
{"level":"debug","ts":"2021-08-25T16:03:53.889Z","caller":"awsutils/awsutils.go:539","msg":"Found ENI MAC address: 02:72:23:ac:41:1b"}
{"level":"debug","ts":"2021-08-25T16:03:53.891Z","caller":"awsutils/awsutils.go:539","msg":"Found ENI: eni-0488a0403d6fa7e41, MAC 02:72:23:ac:41:1b, device 0"}
{"level":"debug","ts":"2021-08-25T16:03:53.893Z","caller":"ipamd/ipamd.go:557","msg":"Reconcile existing ENI eni-0488a0403d6fa7e41 IP pool"}
{"level":"info","ts":"2021-08-25T16:03:53.894Z","caller":"ipamd/ipamd.go:1239","msg":"Reconcile and skip primary IP 172.16.2.235 on ENI eni-0488a0403d6fa7e41"}
{"level":"debug","ts":"2021-08-25T16:03:53.894Z","caller":"ipamd/ipamd.go:557","msg":"Reconcile existing ENI eni-0488a0403d6fa7e41 IP prefixes"}
{"level":"debug","ts":"2021-08-25T16:03:53.894Z","caller":"ipamd/ipamd.go:1172","msg":"Found prefix pool count 3 for eni eni-0488a0403d6fa7e41\n"}
{"level":"debug","ts":"2021-08-25T16:03:53.894Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.160/28"}
{"level":"info","ts":"2021-08-25T16:03:53.894Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.160/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:03:53.894Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:03:53.894Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.16/28"}
{"level":"info","ts":"2021-08-25T16:03:53.894Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.16/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:03:53.894Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:03:53.894Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.176/28"}
{"level":"info","ts":"2021-08-25T16:03:53.895Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.176/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:03:53.895Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:03:53.895Z","caller":"ipamd/ipamd.go:557","msg":"Successfully Reconciled ENI/IP pool"}
{"level":"debug","ts":"2021-08-25T16:03:53.895Z","caller":"ipamd/ipamd.go:557","msg":"IP/Prefix Address Pool stats: total: 48, assigned: 12, total prefixes: 3"}
{"level":"debug","ts":"2021-08-25T16:03:56.395Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:04:01.541Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:04:06.541Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:04:11.542Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:04:16.766Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
^[[1;2D{"level":"debug","ts":"2021-08-25T16:04:21.766Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:04:26.855Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:04:31.861Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:04:36.861Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:04:42.019Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:04:47.019Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:04:52.020Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:04:54.520Z","caller":"ipamd/ipamd.go:557","msg":"Reconciling ENI/IP pool info because time since last 1m0.63172729s <= 1m0s"}
{"level":"debug","ts":"2021-08-25T16:04:54.539Z","caller":"ipamd/ipamd.go:1106","msg":"Total number of interfaces found: 1 "}
{"level":"debug","ts":"2021-08-25T16:04:54.539Z","caller":"awsutils/awsutils.go:539","msg":"Found ENI MAC address: 02:72:23:ac:41:1b"}
{"level":"debug","ts":"2021-08-25T16:04:54.542Z","caller":"awsutils/awsutils.go:539","msg":"Found ENI: eni-0488a0403d6fa7e41, MAC 02:72:23:ac:41:1b, device 0"}
{"level":"debug","ts":"2021-08-25T16:04:54.544Z","caller":"ipamd/ipamd.go:557","msg":"Reconcile existing ENI eni-0488a0403d6fa7e41 IP pool"}
{"level":"info","ts":"2021-08-25T16:04:54.544Z","caller":"ipamd/ipamd.go:1239","msg":"Reconcile and skip primary IP 172.16.2.235 on ENI eni-0488a0403d6fa7e41"}
{"level":"debug","ts":"2021-08-25T16:04:54.544Z","caller":"ipamd/ipamd.go:557","msg":"Reconcile existing ENI eni-0488a0403d6fa7e41 IP prefixes"}
{"level":"debug","ts":"2021-08-25T16:04:54.544Z","caller":"ipamd/ipamd.go:1172","msg":"Found prefix pool count 3 for eni eni-0488a0403d6fa7e41\n"}
{"level":"debug","ts":"2021-08-25T16:04:54.544Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.160/28"}
{"level":"info","ts":"2021-08-25T16:04:54.544Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.160/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:04:54.544Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:04:54.545Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.16/28"}
{"level":"info","ts":"2021-08-25T16:04:54.545Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.16/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:04:54.545Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:04:54.545Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.176/28"}
{"level":"info","ts":"2021-08-25T16:04:54.545Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.176/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:04:54.545Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:04:54.545Z","caller":"ipamd/ipamd.go:557","msg":"Successfully Reconciled ENI/IP pool"}
{"level":"debug","ts":"2021-08-25T16:04:54.545Z","caller":"ipamd/ipamd.go:557","msg":"IP/Prefix Address Pool stats: total: 48, assigned: 12, total prefixes: 3"}
{"level":"debug","ts":"2021-08-25T16:04:57.045Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:05:02.239Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:05:07.239Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:05:12.239Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:05:17.240Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:05:22.692Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:05:27.739Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:05:32.740Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:05:37.740Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:05:42.740Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:05:47.741Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:05:53.319Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:05:55.819Z","caller":"ipamd/ipamd.go:557","msg":"Reconciling ENI/IP pool info because time since last 1m1.298845694s <= 1m0s"}
{"level":"debug","ts":"2021-08-25T16:05:55.820Z","caller":"ipamd/ipamd.go:1106","msg":"Total number of interfaces found: 1 "}
{"level":"debug","ts":"2021-08-25T16:05:55.820Z","caller":"awsutils/awsutils.go:539","msg":"Found ENI MAC address: 02:72:23:ac:41:1b"}
{"level":"debug","ts":"2021-08-25T16:05:55.822Z","caller":"awsutils/awsutils.go:539","msg":"Found ENI: eni-0488a0403d6fa7e41, MAC 02:72:23:ac:41:1b, device 0"}
{"level":"debug","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:557","msg":"Reconcile existing ENI eni-0488a0403d6fa7e41 IP pool"}
{"level":"info","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:1239","msg":"Reconcile and skip primary IP 172.16.2.235 on ENI eni-0488a0403d6fa7e41"}
{"level":"debug","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:557","msg":"Reconcile existing ENI eni-0488a0403d6fa7e41 IP prefixes"}
{"level":"debug","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:1172","msg":"Found prefix pool count 3 for eni eni-0488a0403d6fa7e41\n"}
{"level":"debug","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.160/28"}
{"level":"info","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.160/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.16/28"}
{"level":"info","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.16/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:1281","msg":"Check in coolddown Found prefix 172.16.2.176/28"}
{"level":"info","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:1435","msg":"Adding 172.16.2.176/28 to DS for eni-0488a0403d6fa7e41"}
{"level":"info","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:1435","msg":"IP already in DS"}
{"level":"debug","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:557","msg":"Successfully Reconciled ENI/IP pool"}
{"level":"debug","ts":"2021-08-25T16:05:55.826Z","caller":"ipamd/ipamd.go:557","msg":"IP/Prefix Address Pool stats: total: 48, assigned: 12, total prefixes: 3"}
{"level":"debug","ts":"2021-08-25T16:05:58.326Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}

==> /var/log/aws-routed-eni/plugin.log <==
{"level":"info","ts":"2021-08-25T16:06:00.488Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(ac45ad5d2891cde20800cec3deffff73bb548dfabe12933f0d2e164505ca0a48) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165114-cj56n;K8S_POD_INFRA_CONTAINER_ID=ac45ad5d2891cde20800cec3deffff73bb548dfabe12933f0d2e164505ca0a48) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}

==> /var/log/aws-routed-eni/ipamd.log <==
{"level":"info","ts":"2021-08-25T16:06:00.490Z","caller":"rpc/rpc.pb.go:519","msg":"Received DelNetwork for Sandbox ac45ad5d2891cde20800cec3deffff73bb548dfabe12933f0d2e164505ca0a48"}
{"level":"debug","ts":"2021-08-25T16:06:00.490Z","caller":"rpc/rpc.pb.go:519","msg":"DelNetworkRequest: ClientVersion:\"v1.9.0\" K8S_POD_NAME:\"md-conqueror-satellite-de-04-cron-27165114-cj56n\" K8S_POD_NAMESPACE:\"staging-md-conqueror-satellite-3674\" K8S_POD_INFRA_CONTAINER_ID:\"ac45ad5d2891cde20800cec3deffff73bb548dfabe12933f0d2e164505ca0a48\" Reason:\"PodDeleted\" ContainerID:\"ac45ad5d2891cde20800cec3deffff73bb548dfabe12933f0d2e164505ca0a48\" IfName:\"eth0\" NetworkName:\"aws-cni\" "}
{"level":"debug","ts":"2021-08-25T16:06:00.490Z","caller":"ipamd/rpc_handler.go:204","msg":"UnassignPodIPv4Address: IP address pool stats: total:48, assigned 12, sandbox aws-cni/ac45ad5d2891cde20800cec3deffff73bb548dfabe12933f0d2e164505ca0a48/eth0"}
{"level":"debug","ts":"2021-08-25T16:06:00.490Z","caller":"ipamd/rpc_handler.go:204","msg":"UnassignPodIPv4Address: Failed to find IPAM entry under full key, trying CRI-migrated version"}
{"level":"warn","ts":"2021-08-25T16:06:00.490Z","caller":"ipamd/rpc_handler.go:204","msg":"UnassignPodIPv4Address: Failed to find sandbox _migrated-from-cri/ac45ad5d2891cde20800cec3deffff73bb548dfabe12933f0d2e164505ca0a48/unknown"}
{"level":"info","ts":"2021-08-25T16:06:00.490Z","caller":"rpc/rpc.pb.go:519","msg":"Send DelNetworkReply: IPv4Addr , DeviceNumber: 0, err: datastore: unknown pod"}

==> /var/log/aws-routed-eni/plugin.log <==
{"level":"info","ts":"2021-08-25T16:06:00.490Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Container ac45ad5d2891cde20800cec3deffff73bb548dfabe12933f0d2e164505ca0a48 not found"}
{"level":"info","ts":"2021-08-25T16:06:02.036Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(26da684891b29d95b2e2094b602b8dbed0368528267ee48c072db5450c035c86) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-01-cron-27165114-ptrt4;K8S_POD_INFRA_CONTAINER_ID=26da684891b29d95b2e2094b602b8dbed0368528267ee48c072db5450c035c86) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}

==> /var/log/aws-routed-eni/ipamd.log <==
{"level":"info","ts":"2021-08-25T16:06:02.040Z","caller":"rpc/rpc.pb.go:519","msg":"Received DelNetwork for Sandbox 26da684891b29d95b2e2094b602b8dbed0368528267ee48c072db5450c035c86"}
{"level":"debug","ts":"2021-08-25T16:06:02.041Z","caller":"rpc/rpc.pb.go:519","msg":"DelNetworkRequest: ClientVersion:\"v1.9.0\" K8S_POD_NAME:\"md-conqueror-satellite-pl-01-cron-27165114-ptrt4\" K8S_POD_NAMESPACE:\"staging-md-conqueror-satellite-3674\" K8S_POD_INFRA_CONTAINER_ID:\"26da684891b29d95b2e2094b602b8dbed0368528267ee48c072db5450c035c86\" Reason:\"PodDeleted\" ContainerID:\"26da684891b29d95b2e2094b602b8dbed0368528267ee48c072db5450c035c86\" IfName:\"eth0\" NetworkName:\"aws-cni\" "}
{"level":"debug","ts":"2021-08-25T16:06:02.041Z","caller":"ipamd/rpc_handler.go:204","msg":"UnassignPodIPv4Address: IP address pool stats: total:48, assigned 12, sandbox aws-cni/26da684891b29d95b2e2094b602b8dbed0368528267ee48c072db5450c035c86/eth0"}
{"level":"debug","ts":"2021-08-25T16:06:02.041Z","caller":"ipamd/rpc_handler.go:204","msg":"UnassignPodIPv4Address: Failed to find IPAM entry under full key, trying CRI-migrated version"}
{"level":"warn","ts":"2021-08-25T16:06:02.041Z","caller":"ipamd/rpc_handler.go:204","msg":"UnassignPodIPv4Address: Failed to find sandbox _migrated-from-cri/26da684891b29d95b2e2094b602b8dbed0368528267ee48c072db5450c035c86/unknown"}
{"level":"info","ts":"2021-08-25T16:06:02.041Z","caller":"rpc/rpc.pb.go:519","msg":"Send DelNetworkReply: IPv4Addr , DeviceNumber: 0, err: datastore: unknown pod"}

==> /var/log/aws-routed-eni/plugin.log <==
{"level":"info","ts":"2021-08-25T16:06:02.041Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Container 26da684891b29d95b2e2094b602b8dbed0368528267ee48c072db5450c035c86 not found"}

==> /var/log/aws-routed-eni/ipamd.log <==
{"level":"debug","ts":"2021-08-25T16:06:03.327Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:06:08.327Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:06:13.327Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:06:18.328Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:06:23.328Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:06:28.329Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:06:33.329Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}
{"level":"debug","ts":"2021-08-25T16:06:38.329Z","caller":"ipamd/ipamd.go:1063","msg":"ENI eni-0488a0403d6fa7e41 cannot be deleted because it is primary"}

At some point it ended up with this:

{"level":"info","ts":"2021-08-25T16:57:31.011Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-01-cron-27165168-2lz2v;K8S_POD_INFRA_CONTAINER_ID=9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:57:31.012Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:57:31.027Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-06-cron-27165168-gsrhn;K8S_POD_INFRA_CONTAINER_ID=ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:57:31.027Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:57:31.055Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165168-cpqr5;K8S_POD_INFRA_CONTAINER_ID=8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:57:31.056Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:57:51.239Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-01-cron-27165168-2lz2v;K8S_POD_INFRA_CONTAINER_ID=9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:57:51.240Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:57:51.504Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165168-cpqr5;K8S_POD_INFRA_CONTAINER_ID=4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:57:51.508Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:57:51.524Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-03-cron-27165168-pbzkb;K8S_POD_INFRA_CONTAINER_ID=fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"info","ts":"2021-08-25T16:57:51.524Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-06-cron-27165168-gsrhn;K8S_POD_INFRA_CONTAINER_ID=ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:57:51.525Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"error","ts":"2021-08-25T16:57:51.527Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:57:51.578Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165168-cpqr5;K8S_POD_INFRA_CONTAINER_ID=8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:57:51.581Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:57:52.628Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=redis-master-0;K8S_POD_INFRA_CONTAINER_ID=b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:57:52.632Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:57:52.634Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-01-cron-27165168-2lz2v;K8S_POD_INFRA_CONTAINER_ID=9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"info","ts":"2021-08-25T16:57:52.639Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165168-cpqr5;K8S_POD_INFRA_CONTAINER_ID=4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:57:52.640Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"error","ts":"2021-08-25T16:57:52.641Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:57:52.655Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-06-cron-27165168-gsrhn;K8S_POD_INFRA_CONTAINER_ID=ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:57:52.657Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:57:52.675Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-03-cron-27165168-pbzkb;K8S_POD_INFRA_CONTAINER_ID=fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:57:52.676Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:57:52.700Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165168-cpqr5;K8S_POD_INFRA_CONTAINER_ID=8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:57:52.701Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:57:53.700Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-06-cron-27165168-gsrhn;K8S_POD_INFRA_CONTAINER_ID=ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"info","ts":"2021-08-25T16:57:53.700Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=redis-master-0;K8S_POD_INFRA_CONTAINER_ID=b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:57:53.701Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"error","ts":"2021-08-25T16:57:53.702Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:57:53.737Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-01-cron-27165168-2lz2v;K8S_POD_INFRA_CONTAINER_ID=9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:57:53.739Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:06.041Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165168-cpqr5;K8S_POD_INFRA_CONTAINER_ID=4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:06.046Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:06.086Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165168-cpqr5;K8S_POD_INFRA_CONTAINER_ID=8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:06.087Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:07.025Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=redis-master-0;K8S_POD_INFRA_CONTAINER_ID=b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:07.025Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:07.040Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-03-cron-27165168-pbzkb;K8S_POD_INFRA_CONTAINER_ID=fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:07.040Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:08.038Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-01-cron-27165168-2lz2v;K8S_POD_INFRA_CONTAINER_ID=9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:08.039Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:09.022Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-06-cron-27165168-gsrhn;K8S_POD_INFRA_CONTAINER_ID=ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:09.028Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:17.023Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165168-cpqr5;K8S_POD_INFRA_CONTAINER_ID=4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:17.024Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:17.042Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165168-cpqr5;K8S_POD_INFRA_CONTAINER_ID=8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:17.043Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:18.032Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-03-cron-27165168-pbzkb;K8S_POD_INFRA_CONTAINER_ID=fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:18.033Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:20.035Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=redis-master-0;K8S_POD_INFRA_CONTAINER_ID=b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:20.036Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:21.022Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-01-cron-27165168-2lz2v;K8S_POD_INFRA_CONTAINER_ID=9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:21.023Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:24.063Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-06-cron-27165168-gsrhn;K8S_POD_INFRA_CONTAINER_ID=ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:24.064Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:30.025Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165168-cpqr5;K8S_POD_INFRA_CONTAINER_ID=4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:30.027Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:30.047Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165168-cpqr5;K8S_POD_INFRA_CONTAINER_ID=8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:30.048Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:31.021Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-03-cron-27165168-pbzkb;K8S_POD_INFRA_CONTAINER_ID=fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:31.022Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container fc57e7b3edc3ca8ea781a82d4fc8fb7dc6649375d40ad03f2afaea54644011b1: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:31.409Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-01-cron-27165168-2lz2v;K8S_POD_INFRA_CONTAINER_ID=9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:31.410Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:31.431Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-06-cron-27165168-gsrhn;K8S_POD_INFRA_CONTAINER_ID=ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:31.431Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:31.448Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165168-cpqr5;K8S_POD_INFRA_CONTAINER_ID=8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:31.449Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:32.049Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=redis-master-0;K8S_POD_INFRA_CONTAINER_ID=b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:32.052Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container b85a01e25479f7c87024a621ed5d05c56780c37886fe47e841f58e6a7b0da527: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:35.023Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-06-cron-27165168-gsrhn;K8S_POD_INFRA_CONTAINER_ID=ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:35.024Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container ababd76e130b87035c175523e56450ae3c94041e4da404abe3235099e903b51d: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:36.032Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-pl-01-cron-27165168-2lz2v;K8S_POD_INFRA_CONTAINER_ID=9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:36.033Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 9c864709df2b73c732fc802ce25dc9ba842acd0d8a488419a0307524d8ce70af: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:41.021Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165168-cpqr5;K8S_POD_INFRA_CONTAINER_ID=4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:41.022Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 4366d3c749d810b9019857b5447f8e70aed04a540caa61ac120824a771f07251: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}
{"level":"info","ts":"2021-08-25T16:58:41.037Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Received CNI del request: ContainerID(8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=staging-md-conqueror-satellite-3674;K8S_POD_NAME=md-conqueror-satellite-de-04-cron-27165168-cpqr5;K8S_POD_INFRA_CONTAINER_ID=8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"error","ts":"2021-08-25T16:58:41.038Z","caller":"routed-eni-cni-plugin/cni.go:240","msg":"Error received from DelNetwork gRPC call for container 8b50bfe22b52bbce96f409be1affe436828b7f31181274de3c143772912bf6c3: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused\""}

What you expected to happen: Passing probes, normal operation.

How to reproduce it (as minimally and precisely as possible): I’m afraid I could not provide any

Anything else we need to know?: We are running CronJobs every minute and I had an idea that it could be an issue? Maybe we should adjust some settings of CNI for such utilization?

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.2-eks-0389ca3", GitCommit:"8a4e27b9d88142bbdd21b997b532eb6d493df6d2", GitTreeState:"clean", BuildDate:"2021-07-31T01:34:46Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

• CNI Version v1.9.0-eksbuild.1
• OS (e.g: cat /etc/os-release):

- NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

• Kernel (e.g. uname -a):

Linux ip-172-16-2-235.eu-west-1.compute.internal 5.4.129-63.229.amzn2.x86_64 #1 SMP Tue Jul 20 21:22:08 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux