cilium: Unable to create pods when CNP has both port and port name

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Hi there, I have been scratching my head over this error for the last couple of days:

Events:
  Type     Reason                  Age    From               Message
  ----     ------                  ----   ----               -------
  Normal   Scheduled               5m20s  default-scheduler  Successfully assigned ecommerce-service/ecommerce-service-8478d4bb54-4lkq8 to ip-10-11-68-246.eu-central-1.compute.internal
  Warning  FailedCreatePodSandBox  3m49s  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "72fc3622f173b36a5e51696053e9910991c667fddf07357cbad308b3750e92d4": plugin type="cilium-cni" name="cilium" failed (add): unable to create endpoint: Cilium API client timeout exceeded
  Warning  FailedCreatePodSandBox  2m6s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "688f0db5accd18e3a2d91d048aa4ba8b1e59682f28be021d488a010155c7deab": plugin type="cilium-cni" name="cilium" failed (add): unable to create endpoint: Cilium API client timeout exceeded
  Warning  FailedCreatePodSandBox  21s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "39a7e68646f89d9f1a07557709123d1f0b11f7fc67dd7f94d93d99f97fa3f657": plugin type="cilium-cni" name="cilium" failed (add): unable to create endpoint: Cilium API client timeout exceeded

It started happening when I enabled CiliumNetworkPolicy for Endpoints in a specific namespace. Immediately after I apply the policy, everything works as expected. The problem arises when the Pods are restarted. New pods created after the policy has been applied stay in Pending state forever. Removing the CNP solves the issue. Here is an example policy:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: service-a
  namespace: service-a
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: service-a
      app.kubernetes.io/instance: service-a
  ingress:
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: service-a
    - fromEntities:
        - host
      toPorts:
        - ports:
            - port: "http"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/management/healthcheck"
    - fromEntities:
        - host
      toPorts:
        - ports:
            - port: "9187"
              protocol: TCP
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: "monitoring"
      toPorts:
        - ports:
            - port: "8081"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/metrics"
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: "ingress-nginx"
      toPorts:
        - ports:
            - port: "3000"
              protocol: TCP
          rules: {}
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: service-b
      toPorts:
        - ports:
            - port: "3000"
              protocol: TCP
          rules: {}
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: service-c
      toPorts:
        - ports:
            - port: "3000"
              protocol: TCP
          rules: {}
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: service-a
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: tempo
            k8s:app.kubernetes.io/name: tempo
      toPorts:
        - ports:
            - port: "14268"
              protocol: TCP
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: opentelemetry
            k8s:app.kubernetes.io/name: default-collector
      toPorts:
        - ports:
            - port: "14268"
              protocol: TCP
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: service-b
            k8s:app.kubernetes.io/name: service-b
      toPorts:
        - ports:
            - port: "3000"
              protocol: TCP
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s:k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
          rules:
            dns:
              - matchPattern: "*"
    - toFQDNs:
        - matchName: "sts.eu-central-1.amazonaws.com"
        - matchName: "sns.eu-central-1.amazonaws.com"
        - matchName: "sqs.eu-central-1.amazonaws.com"
        - matchName: "s3.eu-central-1.amazonaws.com"
        - matchName: "auth.europe-west1.gcp.xxxx.com"
        - matchName: "api.europe-west1.gcp.xxxx.com"
        - matchName: "api.xxxxx.com"
    - toCIDR:
        - "169.254.169.254/32"

Why would this prevent the cilium-cni to access the Cilium API?

Cilium Version

Image versions         cilium             quay.io/cilium/cilium:v1.14.1@sha256:edc1d05ea1365c4a8f6ac6982247d5c145181704894bb698619c3827b6963a72: 5
                       hubble-relay       quay.io/cilium/hubble-relay:v1.14.1@sha256:db30e85a7abc10589ce2a97d61ee18696a03dc5ea04d44b4d836d88bd75b59d8: 1
                       hubble-ui          quay.io/cilium/hubble-ui:v0.12.0@sha256:1c876cfa1d5e35bc91e1025c9314f922041592a88b03313c22c1f97a5d2ba88f: 1
                       hubble-ui          quay.io/cilium/hubble-ui-backend:v0.12.0@sha256:8a79a1aad4fc9c2aa2b3e4379af0af872a89fcec9d99e117188190671c66fc2e: 1
                       cilium-operator    quay.io/cilium/operator-generic:v1.14.1@sha256:e061de0a930534c7e3f8feda8330976367971238ccafff42659f104effd4b5f7: 2

Kernel Version

NAME                                            STATUS   ROLES    AGE     VERSION               INTERNAL-IP    EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                   CONTAINER-RUNTIME
ip-10-11-64-135.eu-central-1.compute.internal   Ready    <none>   40d     v1.27.3-eks-a5565ad   10.11.64.135   <none>        Amazon Linux 2   5.10.184-175.731.amzn2.aarch64   containerd://1.6.19
ip-10-11-67-230.eu-central-1.compute.internal   Ready    <none>   6d2h    v1.27.3-eks-a5565ad   10.11.67.230   <none>        Amazon Linux 2   5.10.186-179.751.amzn2.x86_64    containerd://1.6.19
ip-10-11-69-105.eu-central-1.compute.internal   Ready    <none>   3h47m   v1.27.4-eks-8ccc7ba   10.11.69.105   <none>        Amazon Linux 2   5.10.186-179.751.amzn2.aarch64   containerd://1.6.19
ip-10-11-69-71.eu-central-1.compute.internal    Ready    <none>   3h1m    v1.27.4-eks-8ccc7ba   10.11.69.71    <none>        Amazon Linux 2   5.10.186-179.751.amzn2.aarch64   containerd://1.6.19
ip-10-11-75-29.eu-central-1.compute.internal    Ready    <none>   6d15h   v1.27.3-eks-a5565ad   10.11.75.29    <none>        Amazon Linux 2   5.10.184-175.731.amzn2.aarch64   containerd://1.6.19
ip-10-11-77-129.eu-central-1.compute.internal   Ready    <none>   23h     v1.27.3-eks-a5565ad   10.11.77.129   <none>        Amazon Linux 2   5.10.186-179.751.amzn2.aarch64   containerd://1.6.19

Kubernetes Version

See above

Sysdump

Can’t share that. Contains sensitive information.

Relevant log output

Common labels: {"app":"cilium-agent","container":"cilium-agent","namespace":"cilium","stream":"stderr"}
Line limit: "3000 (69 returned)"
Total bytes processed: "928  kB"


2023-08-30T08:59:58+02:00	level=error msg="endpoint regeneration failed" containerID=d256d8f937 datapathPolicyRevision=11 desiredPolicyRevision=11 endpointID=2945 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=63362 ipv4=10.11.72.9 ipv6="fe80::e889:76ff:feb4:9c5c" k8sPodName=pharmacy-app-web/pharmacy-app-web-7f8b554c8-xfvjs subsys=endpoint
2023-08-30T08:59:58+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=d256d8f937 datapathPolicyRevision=11 desiredPolicyRevision=11 endpointID=2945 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=63362 ipv4=10.11.72.9 ipv6="fe80::e889:76ff:feb4:9c5c" k8sPodName=pharmacy-app-web/pharmacy-app-web-7f8b554c8-xfvjs mapSync="26.314µs" policyCalculation="37.833µs" prepareBuild="81.279µs" proxyConfiguration="7.122µs" proxyPolicyCalculation="21.21µs" proxyWaitForAck="869.74µs" reason="one or more identities created or deleted" subsys=endpoint total=2.852582ms waitingForCTClean=229ns waitingForLock=969ns
2023-08-30T08:59:58+02:00	level=error msg="endpoint regeneration failed" containerID=633c9e5898 datapathPolicyRevision=11 desiredPolicyRevision=11 endpointID=395 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=57992 ipv4=10.11.68.23 ipv6="fe80::4cd5:10ff:fe40:d41d" k8sPodName=shipcloud-gateway-service/shipcloud-gateway-service-6f8b668567-rkskp subsys=endpoint
2023-08-30T08:59:58+02:00	level=error msg="endpoint regeneration failed" containerID=18d0035c54 datapathPolicyRevision=11 desiredPolicyRevision=11 endpointID=791 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=3558 ipv4=10.11.72.5 ipv6="fe80::e87b:71ff:fe30:a71" k8sPodName=order-service/order-service-74b46d56c4-7mq5c subsys=endpoint
2023-08-30T08:59:58+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=633c9e5898 datapathPolicyRevision=11 desiredPolicyRevision=11 endpointID=395 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=57992 ipv4=10.11.68.23 ipv6="fe80::4cd5:10ff:fe40:d41d" k8sPodName=shipcloud-gateway-service/shipcloud-gateway-service-6f8b668567-rkskp mapSync="26.445µs" policyCalculation="33.846µs" prepareBuild="84.946µs" proxyConfiguration="6.753µs" proxyPolicyCalculation="17.509µs" proxyWaitForAck="817.737µs" reason="one or more identities created or deleted" subsys=endpoint total=2.672251ms waitingForCTClean=246ns waitingForLock=814ns
2023-08-30T08:59:58+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=18d0035c54 datapathPolicyRevision=11 desiredPolicyRevision=11 endpointID=791 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=3558 ipv4=10.11.72.5 ipv6="fe80::e87b:71ff:fe30:a71" k8sPodName=order-service/order-service-74b46d56c4-7mq5c mapSync="32.049µs" policyCalculation="28.89µs" prepareBuild="79.425µs" proxyConfiguration="6.243µs" proxyPolicyCalculation="16.722µs" proxyWaitForAck="958.485µs" reason="one or more identities created or deleted" subsys=endpoint total=2.760086ms waitingForCTClean=287ns waitingForLock=485ns
2023-08-30T08:59:58+02:00	level=warning msg="Failed to apply L7 proxy policy changes. These will be re-applied in future updates." error="context cancelled before waiting for proxy updates: context canceled" subsys=endpoint-manager
2023-08-30T08:59:58+02:00	level=warning msg="NACK received for versions after 116 and up to 117; waiting for a version update before sending again" subsys=xds xdsAckedVersion=116 xdsClientNode="host~127.0.0.1~no-id~localdomain" xdsDetail="PortNetworkPolicy: Duplicate port number 3000" xdsNonce=117 xdsStreamID=3 xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy
2023-08-30T08:59:58+02:00	level=warning msg="[gRPC config for type.googleapis.com/cilium.NetworkPolicy rejected: PortNetworkPolicy: Duplicate port number 3000" subsys=envoy-config threadID=499
2023-08-30T08:59:58+02:00	level=warning msg="UpdateIdentities: Skipping Delete of a non-existing identity" identity=16777222 subsys=policy
2023-08-30T08:59:58+02:00	level=warning msg="UpdateIdentities: Skipping Delete of a non-existing identity" identity=16777220 subsys=policy
2023-08-30T08:59:58+02:00	level=warning msg="UpdateIdentities: Skipping Delete of a non-existing identity" identity=16777221 subsys=policy
2023-08-30T08:59:58+02:00	level=warning msg="UpdateIdentities: Skipping Delete of a non-existing identity" identity=16777221 subsys=policy
2023-08-30T08:59:58+02:00	level=warning msg="UpdateIdentities: Skipping Delete of a non-existing identity" identity=16777219 subsys=policy
2023-08-30T08:59:58+02:00	level=warning msg="UpdateIdentities: Skipping Delete of a non-existing identity" identity=16777219 subsys=policy
2023-08-30T08:59:47+02:00	level=error msg="endpoint regeneration failed" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d subsys=endpoint
2023-08-30T08:59:47+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=49.869508ms bpfWaitForELF="3.528µs" bpfWriteELF="643.289µs" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d mapSync="80.146µs" policyCalculation="18.068µs" prepareBuild="408.848µs" proxyConfiguration="122.017µs" proxyPolicyCalculation="137.91µs" proxyWaitForAck="2.732µs" reason="retrying regeneration" subsys=endpoint total=52.587282ms waitingForCTClean=328ns waitingForLock="1.289µs"
2023-08-30T08:59:47+02:00	level=warning msg="NACK received for versions after 114 and up to 115; waiting for a version update before sending again" subsys=xds xdsAckedVersion=114 xdsClientNode="host~127.0.0.1~no-id~localdomain" xdsDetail="PortNetworkPolicy: Duplicate port number 3000" xdsNonce=115 xdsStreamID=3 xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy
2023-08-30T08:59:47+02:00	level=warning msg="[gRPC config for type.googleapis.com/cilium.NetworkPolicy rejected: PortNetworkPolicy: Duplicate port number 3000" subsys=envoy-config threadID=499
2023-08-30T08:59:37+02:00	level=error msg="endpoint regeneration failed" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d subsys=endpoint
2023-08-30T08:59:37+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=49.454006ms bpfWaitForELF="3.569µs" bpfWriteELF="593.009µs" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d mapSync="79.655µs" policyCalculation="18.273µs" prepareBuild="434.35µs" proxyConfiguration="118.743µs" proxyPolicyCalculation="141.324µs" proxyWaitForAck="3.044µs" reason="retrying regeneration" subsys=endpoint total=52.075633ms waitingForCTClean=279ns waitingForLock="1.174µs"
2023-08-30T08:59:37+02:00	level=warning msg="NACK received for versions after 112 and up to 113; waiting for a version update before sending again" subsys=xds xdsAckedVersion=112 xdsClientNode="host~127.0.0.1~no-id~localdomain" xdsDetail="PortNetworkPolicy: Duplicate port number 3000" xdsNonce=113 xdsStreamID=3 xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy
2023-08-30T08:59:37+02:00	level=warning msg="[gRPC config for type.googleapis.com/cilium.NetworkPolicy rejected: PortNetworkPolicy: Duplicate port number 3000" subsys=envoy-config threadID=499
2023-08-30T08:59:29+02:00	level=error msg="endpoint regeneration failed" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d subsys=endpoint
2023-08-30T08:59:29+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=49.488624ms bpfWaitForELF="3.742µs" bpfWriteELF="557.703µs" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d mapSync="88.623µs" policyCalculation="20.299µs" prepareBuild="410.529µs" proxyConfiguration="128.712µs" proxyPolicyCalculation="136.868µs" proxyWaitForAck="2.674µs" reason="retrying regeneration" subsys=endpoint total=52.146942ms waitingForCTClean=304ns waitingForLock="1.608µs"
2023-08-30T08:59:29+02:00	level=warning msg="NACK received for versions after 110 and up to 111; waiting for a version update before sending again" subsys=xds xdsAckedVersion=110 xdsClientNode="host~127.0.0.1~no-id~localdomain" xdsDetail="PortNetworkPolicy: Duplicate port number 3000" xdsNonce=111 xdsStreamID=3 xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy
2023-08-30T08:59:29+02:00	level=warning msg="[gRPC config for type.googleapis.com/cilium.NetworkPolicy rejected: PortNetworkPolicy: Duplicate port number 3000" subsys=envoy-config threadID=499
2023-08-30T08:59:23+02:00	level=error msg="endpoint regeneration failed" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d subsys=endpoint
2023-08-30T08:59:23+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=49.777186ms bpfWaitForELF="4.118µs" bpfWriteELF="577.33µs" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d mapSync="78.908µs" policyCalculation="17.706µs" prepareBuild="401.291µs" proxyConfiguration="120.426µs" proxyPolicyCalculation="136.933µs" proxyWaitForAck="2.617µs" reason="retrying regeneration" subsys=endpoint total=52.392067ms waitingForCTClean=361ns waitingForLock="1.272µs"
2023-08-30T08:59:23+02:00	level=warning msg="NACK received for versions after 108 and up to 109; waiting for a version update before sending again" subsys=xds xdsAckedVersion=108 xdsClientNode="host~127.0.0.1~no-id~localdomain" xdsDetail="PortNetworkPolicy: Duplicate port number 3000" xdsNonce=109 xdsStreamID=3 xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy
2023-08-30T08:59:23+02:00	level=warning msg="[gRPC config for type.googleapis.com/cilium.NetworkPolicy rejected: PortNetworkPolicy: Duplicate port number 3000" subsys=envoy-config threadID=499
2023-08-30T08:59:19+02:00	level=error msg="endpoint regeneration failed" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d subsys=endpoint
2023-08-30T08:59:19+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=49.353323ms bpfWaitForELF="3.996µs" bpfWriteELF="571.307µs" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d mapSync="77.365µs" policyCalculation="17.715µs" prepareBuild="406.19µs" proxyConfiguration="114.904µs" proxyPolicyCalculation="148.084µs" proxyWaitForAck="3.093µs" reason="retrying regeneration" subsys=endpoint total=51.949038ms waitingForCTClean=337ns waitingForLock="1.198µs"
2023-08-30T08:59:19+02:00	level=warning msg="NACK received for versions after 106 and up to 107; waiting for a version update before sending again" subsys=xds xdsAckedVersion=106 xdsClientNode="host~127.0.0.1~no-id~localdomain" xdsDetail="PortNetworkPolicy: Duplicate port number 3000" xdsNonce=107 xdsStreamID=3 xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy
2023-08-30T08:59:19+02:00	level=warning msg="[gRPC config for type.googleapis.com/cilium.NetworkPolicy rejected: PortNetworkPolicy: Duplicate port number 3000" subsys=envoy-config threadID=499
2023-08-30T08:59:17+02:00	level=error msg="endpoint regeneration failed" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d subsys=endpoint
2023-08-30T08:59:17+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=202.068752ms bpfWaitForELF="3.569µs" bpfWriteELF="589.112µs" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d mapSync="75.577µs" policyCalculation="13.735µs" prepareBuild="377.586µs" proxyConfiguration="111.49µs" proxyPolicyCalculation="270.051µs" proxyWaitForAck="2.215µs" reason="retrying regeneration" subsys=endpoint total=204.61783ms waitingForCTClean=370ns waitingForLock="1.009µs"
2023-08-30T08:59:17+02:00	level=warning msg="[gRPC config for type.googleapis.com/cilium.NetworkPolicy rejected: PortNetworkPolicy: Duplicate port number 3000" subsys=envoy-config threadID=499
2023-08-30T08:59:17+02:00	level=warning msg="NACK received for versions after 104 and up to 105; waiting for a version update before sending again" subsys=xds xdsAckedVersion=104 xdsClientNode="host~127.0.0.1~no-id~localdomain" xdsDetail="PortNetworkPolicy: Duplicate port number 3000" xdsNonce=105 xdsStreamID=3 xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy
2023-08-30T08:59:17+02:00	level=error msg="endpoint regeneration failed" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d subsys=endpoint
2023-08-30T08:59:17+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d mapSync="83.453µs" policyCalculation="13.891µs" prepareBuild="81.573µs" proxyConfiguration="95.359µs" proxyPolicyCalculation="117.897µs" proxyWaitForAck=87.858307ms reason="one or more identities created or deleted" subsys=endpoint total=88.896078ms waitingForCTClean=361ns waitingForLock=745ns
2023-08-30T08:59:17+02:00	level=warning msg="NACK received for versions after 102 and up to 103; waiting for a version update before sending again" subsys=xds xdsAckedVersion=102 xdsClientNode="host~127.0.0.1~no-id~localdomain" xdsDetail="PortNetworkPolicy: Duplicate port number 3000" xdsNonce=103 xdsStreamID=3 xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy
2023-08-30T08:59:17+02:00	level=warning msg="[gRPC config for type.googleapis.com/cilium.NetworkPolicy rejected: PortNetworkPolicy: Duplicate port number 3000" subsys=envoy-config threadID=499
2023-08-30T08:59:17+02:00	level=error msg="endpoint regeneration failed" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d subsys=endpoint
2023-08-30T08:59:17+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=512.2881ms bpfWaitForELF="4.168µs" bpfWriteELF="592.369µs" containerID=8737350d1d datapathPolicyRevision=9 desiredPolicyRevision=10 endpointID=3669 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=37982 ipv4=10.11.72.15 ipv6="fe80::7c4b:88ff:fea4:633d" k8sPodName=ecommerce-service/ecommerce-service-685957d596-87p6d mapSync="733.267µs" policyCalculation=3.175908ms prepareBuild="398.337µs" proxyConfiguration="202.771µs" proxyPolicyCalculation="114.87µs" proxyWaitForAck="2.987µs" reason="policy rules added" subsys=endpoint total=518.916564ms waitingForCTClean=344ns waitingForLock="1.247µs"
2023-08-30T08:59:16+02:00	level=error msg="endpoint regeneration failed" containerID=f9a03db363 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=883 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=4399 ipv4=10.11.72.6 ipv6="fe80::ecdc:50ff:fe7d:1ffb" k8sPodName=tempo/tempo-gateway-767676c577-xv9zn subsys=endpoint
2023-08-30T08:59:16+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=f9a03db363 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=883 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=4399 ipv4=10.11.72.6 ipv6="fe80::ecdc:50ff:fe7d:1ffb" k8sPodName=tempo/tempo-gateway-767676c577-xv9zn mapSync=1.169413ms policyCalculation="33.517µs" prepareBuild="67.847µs" proxyConfiguration="29.793µs" proxyPolicyCalculation="348.024µs" proxyWaitForAck="1.387µs" reason="one or more identities created or deleted" subsys=endpoint total=1.989889ms waitingForCTClean=296ns waitingForLock=631ns
2023-08-30T08:59:16+02:00	level=error msg="endpoint regeneration failed" containerID=036f9d4601 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=396 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=55250 ipv4=10.11.72.10 ipv6="fe80::441d:a4ff:fe58:15d8" k8sPodName=logistic-service/logistic-service-5bb98bb759-n95j7 subsys=endpoint
2023-08-30T08:59:16+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=036f9d4601 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=396 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=55250 ipv4=10.11.72.10 ipv6="fe80::441d:a4ff:fe58:15d8" k8sPodName=logistic-service/logistic-service-5bb98bb759-n95j7 mapSync="80.63µs" policyCalculation="32.968µs" prepareBuild="59.995µs" proxyConfiguration="5.874µs" proxyPolicyCalculation="17.198µs" proxyWaitForAck=1.274141ms reason="one or more identities created or deleted" subsys=endpoint total=2.211441ms waitingForCTClean=246ns waitingForLock=804ns
2023-08-30T08:59:16+02:00	level=error msg="endpoint regeneration failed" containerID=486a52b991 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=2734 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=25882 ipv4=10.11.73.22 ipv6="fe80::eca1:edff:fe82:e808" k8sPodName=fax-service/fax-service-56c5b5c554-p2fgr subsys=endpoint
2023-08-30T08:59:16+02:00	level=error msg="endpoint regeneration failed" containerID=48f1d2ab03 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=1662 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=62294 ipv4=10.11.65.209 ipv6="fe80::88c8:82ff:fe3e:6ffe" k8sPodName=logging/loki-gateway-5dd67b5f79-dnqlc subsys=endpoint
2023-08-30T08:59:16+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=486a52b991 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=2734 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=25882 ipv4=10.11.73.22 ipv6="fe80::eca1:edff:fe82:e808" k8sPodName=fax-service/fax-service-56c5b5c554-p2fgr mapSync="314.096µs" policyCalculation="551.213µs" prepareBuild="75.281µs" proxyConfiguration="6.564µs" proxyPolicyCalculation="45.185µs" proxyWaitForAck="677.292µs" reason="one or more identities created or deleted" subsys=endpoint total=1.738069ms waitingForCTClean=353ns waitingForLock=624ns
2023-08-30T08:59:16+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=48f1d2ab03 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=1662 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=62294 ipv4=10.11.65.209 ipv6="fe80::88c8:82ff:fe3e:6ffe" k8sPodName=logging/loki-gateway-5dd67b5f79-dnqlc mapSync="329.505µs" policyCalculation="547.25µs" prepareBuild="64.821µs" proxyConfiguration="68.455µs" proxyPolicyCalculation="455.017µs" proxyWaitForAck="1.608µs" reason="one or more identities created or deleted" subsys=endpoint total=1.552414ms waitingForCTClean=295ns waitingForLock=665ns
2023-08-30T08:59:16+02:00	level=error msg="endpoint regeneration failed" containerID=d256d8f937 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=2945 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=63362 ipv4=10.11.72.9 ipv6="fe80::e889:76ff:feb4:9c5c" k8sPodName=pharmacy-app-web/pharmacy-app-web-7f8b554c8-xfvjs subsys=endpoint
2023-08-30T08:59:16+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=d256d8f937 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=2945 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=63362 ipv4=10.11.72.9 ipv6="fe80::e889:76ff:feb4:9c5c" k8sPodName=pharmacy-app-web/pharmacy-app-web-7f8b554c8-xfvjs mapSync="72.852µs" policyCalculation="26.297µs" prepareBuild="55.236µs" proxyConfiguration="4.537µs" proxyPolicyCalculation="17.321µs" proxyWaitForAck="718.611µs" reason="one or more identities created or deleted" subsys=endpoint total="972.098µs" waitingForCTClean=221ns waitingForLock=616ns
2023-08-30T08:59:16+02:00	level=error msg="endpoint regeneration failed" containerID=735acad36f datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=3435 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=20517 ipv4=10.11.72.11 ipv6="fe80::c00d:abff:fef7:97d8" k8sPodName=flux-system/notification-controller-5fc859457d-qlb45 subsys=endpoint
2023-08-30T08:59:16+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=735acad36f datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=3435 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=20517 ipv4=10.11.72.11 ipv6="fe80::c00d:abff:fef7:97d8" k8sPodName=flux-system/notification-controller-5fc859457d-qlb45 mapSync="177.434µs" policyCalculation=1.713413ms prepareBuild="69.989µs" proxyConfiguration="88.73µs" proxyPolicyCalculation="233.515µs" proxyWaitForAck="720.737µs" reason="one or more identities created or deleted" subsys=endpoint total=3.071926ms waitingForCTClean=377ns waitingForLock=920ns
2023-08-30T08:59:16+02:00	level=error msg="endpoint regeneration failed" containerID=f1a804cef9 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=227 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=8682 ipv4=10.11.68.29 ipv6="fe80::2c6b:cff:fe16:1c2b" k8sPodName=logging/loki-querier-2 subsys=endpoint
2023-08-30T08:59:16+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=f1a804cef9 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=227 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=8682 ipv4=10.11.68.29 ipv6="fe80::2c6b:cff:fe16:1c2b" k8sPodName=logging/loki-querier-2 mapSync="19.315µs" policyCalculation="30.227µs" prepareBuild="55.949µs" proxyConfiguration="5.751µs" proxyPolicyCalculation="19.126µs" proxyWaitForAck="766.791µs" reason="one or more identities created or deleted" subsys=endpoint total="971.556µs" waitingForCTClean=222ns waitingForLock=705ns
2023-08-30T08:59:16+02:00	level=error msg="endpoint regeneration failed" containerID=69aab45403 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=1746 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=34198 ipv4=10.11.73.25 ipv6="fe80::c802:2dff:fec9:2d7a" k8sPodName=monitoring/prometheus-adapter-7749d778fd-m4jpw subsys=endpoint
2023-08-30T08:59:16+02:00	level=error msg="endpoint regeneration failed" containerID=2f94856dfa datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=1628 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=3612 ipv4=10.11.73.20 ipv6="fe80::881f:86ff:fe57:24c9" k8sPodName=category-tree-service/category-tree-service-769c86df79-jgsjx subsys=endpoint
2023-08-30T08:59:16+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=69aab45403 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=1746 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=34198 ipv4=10.11.73.25 ipv6="fe80::c802:2dff:fec9:2d7a" k8sPodName=monitoring/prometheus-adapter-7749d778fd-m4jpw mapSync="52.159µs" policyCalculation="475.301µs" prepareBuild="64.385µs" proxyConfiguration="6.72µs" proxyPolicyCalculation="456.207µs" proxyWaitForAck="388.171µs" reason="one or more identities created or deleted" subsys=endpoint total=1.521391ms waitingForCTClean=353ns waitingForLock=861ns
2023-08-30T08:59:16+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=2f94856dfa datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=1628 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=3612 ipv4=10.11.73.20 ipv6="fe80::881f:86ff:fe57:24c9" k8sPodName=category-tree-service/category-tree-service-769c86df79-jgsjx mapSync="28.119µs" policyCalculation="36.791µs" prepareBuild="85.586µs" proxyConfiguration="5.71µs" proxyPolicyCalculation="21.407µs" proxyWaitForAck=1.510544ms reason="one or more identities created or deleted" subsys=endpoint total=1.762388ms waitingForCTClean=263ns waitingForLock=853ns
2023-08-30T08:59:16+02:00	level=error msg="endpoint regeneration failed" containerID=efcb500e0c datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=3145 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=42767 ipv4=10.11.72.2 ipv6="fe80::f4de:acff:fee6:cc60" k8sPodName=kube-system/coredns-794954d47b-hwq28 subsys=endpoint
2023-08-30T08:59:16+02:00	level=error msg="endpoint regeneration failed" containerID=901581ee36 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=3203 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=11236 ipv4=10.11.73.28 ipv6="fe80::a860:a3ff:fe6c:74f1" k8sPodName=care-gateway-service/care-gateway-service-f996fbd5b-hmnvs subsys=endpoint
2023-08-30T08:59:16+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=efcb500e0c datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=3145 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=42767 ipv4=10.11.72.2 ipv6="fe80::f4de:acff:fee6:cc60" k8sPodName=kube-system/coredns-794954d47b-hwq28 mapSync="23.377µs" policyCalculation="494.409µs" prepareBuild="64.762µs" proxyConfiguration="6.9µs" proxyPolicyCalculation="24.927µs" proxyWaitForAck="686.924µs" reason="one or more identities created or deleted" subsys=endpoint total=1.381864ms waitingForCTClean=320ns waitingForLock=720ns
2023-08-30T08:59:16+02:00	level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s containerID=901581ee36 datapathPolicyRevision=10 desiredPolicyRevision=10 endpointID=3203 error="Error while configuring proxy redirects: proxy state changes failed: NACK received: PortNetworkPolicy: Duplicate port number 3000" identity=11236 ipv4=10.11.73.28 ipv6="fe80::a860:a3ff:fe6c:74f1" k8sPodName=care-gateway-service/care-gateway-service-f996fbd5b-hmnvs mapSync="26.92µs" policyCalculation="152.687µs" prepareBuild="75.167µs" proxyConfiguration="6.194µs" proxyPolicyCalculation="156.387µs" proxyWaitForAck=1.479258ms reason="one or more identities created or deleted" subsys=endpoint total=2.003854ms waitingForCTClean=328ns waitingForLock=838ns
2023-08-30T08:59:16+02:00	level=warning msg="NACK received for versions after 100 and up to 101; waiting for a version update before sending again" subsys=xds xdsAckedVersion=100 xdsClientNode="host~127.0.0.1~no-id~localdomain" xdsDetail="PortNetworkPolicy: Duplicate port number 3000" xdsNonce=101 xdsStreamID=3 xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy
2023-08-30T08:59:16+02:00	level=warning msg="[gRPC config for type.googleapis.com/cilium.NetworkPolicy rejected: PortNetworkPolicy: Duplicate port number 3000" subsys=envoy-config threadID=499

Anything else?

No response

Code of Conduct

• I agree to follow this project’s Code of Conduct