cilium: transparent encryption: setting nodEncryption=true results in nodes being unreachable

I think the confusion was simply, that we didn’t expect there is a value for a feature that is not functional at all / is for a beta feature despite nothing explicitly stating that.

We’ve recently removed it from docs so there may still be a couple references to it. Sorry about that.

@jrfastab I tried your second suggestion by enabling bpf.masquerade plus I also enabled ipMasqAgent.enabled.

Unfortunately to get bpf.masquerade to work you need to set encryption.enabled = false as “IPSec cannot be used with BPF NodePort” as the logs states and reverts it back to iptables.

level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"enable-host-reachable-services\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
level=warning msg="IPSec cannot be used with BPF NodePort. Disabling BPF NodePort feature." subsys=daemon
level=warning msg="Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host." subsys=daemon
level=info msg="Restored services from maps" failed=0 restored=10 subsys=service
level=info msg="Envoy: Starting xDS gRPC server listening on /var/run/cilium/xds.sock" subsys=envoy-manager
level=info msg="Reading old endpoints..." subsys=daemon
level=info msg="Reusing previous DNS proxy port: 36125" subsys=daemon
level=info msg="Waiting until all Cilium CRDs are available" subsys=k8s
level=info msg="All Cilium CRDs have been found and are available" subsys=k8s
level=info msg="Retrieved node information from kubernetes node" nodeName=ip-10-16-5-189.eu-west-2.compute.internal subsys=k8s
level=info msg="Received own node information from API server" ipAddr.ipv4=10.16.5.189 ipAddr.ipv6="<nil>" k8sNodeIP=10.16.5.189 labels="map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:m5.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-2 failure-domain.beta.kubernetes.io/zone:eu-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-10-16-5-189.eu-west-2.compute.internal kubernetes.io/os:linux node.kubernetes.io/instance-type:m5.xlarge topology.kubernetes.io/region:eu-west-2 topology.kubernetes.io/zone:eu-west-2c workload:trusted]" nodeName=ip-10-16-5-189.eu-west-2.compute.internal subsys=k8s v4Prefix=10.189.0.0/16 v6Prefix="<nil>"
level=info msg="k8s mode: Allowing localhost to reach local endpoints" subsys=daemon
level=info msg="Enabling k8s event listener" subsys=k8s-watcher
level=warning msg="BPF masquerade requires NodePort (--enable-node-port=\"true\"). Falling back to iptables-based masquerading." subsys=daemon
level=fatal msg="BPF ip-masq-agent requires --masquerade=\"true\" and --enable-bpf-masquerade=\"true\"" subsys=daemon

When you have the time, let me know what result you get from your testing.

The documentation for this has been updated to be more clear about what’s supported for now. We would welcome further investigation into enabling this use case if someone is interested in working on it, but we will not block the v1.9.0 release upon this feature.

Proposed document update,

--- a/Documentation/gettingstarted/encryption.rst
+++ b/Documentation/gettingstarted/encryption.rst
@@ -107,6 +107,11 @@ In order to enable node-to-node encryption, add:
     --set encryption.enabled=true \
     --set encryption.nodeEncryption=true
 
+.. note::
+
+    Node to node encryption feature is tested and supported with direct routing
+    modes. Using with tunnel modes is not currently tested or supported.
+

https://github.com/cilium/cilium/pull/13800