cilium: CiliumEndpoint missing for a pod

@christarazi @Weil0ng We should only delete the CEP with the pre condition that its UID is the same when the CEP was created. Something like this:

diff --git a/pkg/k8s/watchers/endpointsynchronizer.go b/pkg/k8s/watchers/endpointsynchronizer.go
index 9e716c85b4..b4fe16a44e 100644
--- a/pkg/k8s/watchers/endpointsynchronizer.go
+++ b/pkg/k8s/watchers/endpointsynchronizer.go
@@ -177,6 +177,9 @@ func (epSync *EndpointSynchronizer) RunK8sCiliumEndpointSync(e *endpoint.Endpoin
                                                        return err
                                                }
 
+                                               // Store CEP UID
+                                               e.CEPUID = localCEP.UID
+
                                                // continue the execution so we update the endpoint
                                                // status immediately upon endpoint creation
                                        case err != nil:
@@ -324,8 +327,15 @@ func deleteCEP(ctx context.Context, scopedLog *logrus.Entry, ciliumClient v2.Cil
                scopedLog.Debug("Skipping CiliumEndpoint deletion because it has no k8s namespace")
                return nil
        }
-       if err := ciliumClient.CiliumEndpoints(namespace).Delete(ctx, podName, meta_v1.DeleteOptions{}); err != nil {
-               if !k8serrors.IsNotFound(err) {
+
+       err := ciliumClient.CiliumEndpoints(namespace).Delete(ctx, podName, meta_v1.DeleteOptions{
+               Preconditions: &meta_v1.Preconditions{
+                       UID: func() *types.UID { a := types.UID(e.CEPUID); return &a }(),
+               },
+       })
+
+       if err != nil {
+               if !k8serrors.IsNotFound(err) || !k8serrors.PreConditionFailed() {
                        scopedLog.WithError(err).Warning("Unable to delete CEP")
                }
        }

The CEP object is only removed because it has the ownerReference set to the backing pod, so that their lifecycles are tied together.

Basically, the timeline of events for an endpoint-delete is:

• kubectl delete pod app1
• kubelet sends CNI DEL to Cilium
• Cilium calls DeleteEndpoint(), which deletes the endpoint internally (removes from BPF maps, etc)
• app1 pod resource is deleted as the CNI DEL has completed
• Cilium’s CEP watcher receives a CEP delete (because of ownerReference set to the now-deleted pod)
  • Deletes endpoint’s IP from ipcache, etc

@Weil0ng

I’m actually still a bit confused as why we would have two pods of same name in this case…as you mentioned k8s is assuming uniqueness of resource names, no? This sounds like a general k8s issue?

I think that’s at the core of my confusion. I think Cilium must follow whatever Kubernetes deems “uniqueness” is. If statefulsets are the exception, then Cilium must account for that.

@liuyuan10 The above is why I’m not saying to just go right ahead re: adding a check for containerID in the delete. I think the solution needs to dig one level deeper.

How does it verify the pod exists without connecting to k8s? Do you mean it restores first and then later verify with k8s and trash the endpoint?

Yes that’s correct.

Because the pod is quickly recreated by k8s while cilium is getting restarted, probably it won’t trash it because there is a pod with the same name.

Correct yeah it seems to me that the pod is back up by the time Cilium goes to validate the restored endpoint.

What’s in question to me is that why the CNI ADD in (5) didn’t add a CEP assuming the it’s already removed at that time.

I think because the CEP already exists with the pod name, because CEPs are named after pod names.

Curious in this issue, a bit confused by above chain of events, so when and who issues the DELETE for the CEP of “app1” to the apiserver before any watcher can get the delete event?

According to christarazi , I think it’s when the k8s pod is removed, apiserver removes the CEP as well because threre is a owner ref to k8s pod.