cilium: Transparent Encryption Not Working

Cilium version (run cilium version) 1.5.4
Kernel version (run uname -a) Linux ip-172-21-53-107 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64 GNU/Linux
Orchestration system version in use (e.g. kubectl version, Mesos, …) kubectl version

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"6f482974b76db3f1e0f5d24605a9d1d38fad9a2b", GitTreeState:"clean", BuildDate:"2019-03-25T02:52:13Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.8", GitCommit:"a89f8c11a5f4f132503edbc4918c98518fd504e3", GitTreeState:"clean", BuildDate:"2019-04-23T04:41:47Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Link to relevant artifacts (policies, deployments scripts, …) NA
Upload a system dump (run curl -sLO releases.cilium.io/tools/cluster-diagnosis.zip && python cluster-diagnosis.zip sysdump and then attach the generated zip file) cilium-sysdump-20190710-134334.zip

How to reproduce the issue

Following the steps on https://cilium.readthedocs.io/en/stable/gettingstarted/encryption after the standard installation.

sample error logs

level=error msg="IPsec enable failed" error="unable to replace local state: invalid argument" local-ip=100.96.0.1/24 reason="local IPv4" remote-ip=0.0.0.0/0 spi=0 subsys=linux-datapath
level=error msg="IPsec enable failed" error="unable to replace remote state: invalid argument" local-ip=100.96.0.1/24 reason=IPv4 remote-ip=100.96.1.1/24 spi=0 subsys=linux-datapath
level=info msg="New encryption keys reclaiming SPI" spi=0 subsys=ipsec

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 24 (11 by maintainers)

Commits related to this issue

cilium: encryption, fix getting started guides create secrects command [ upstream commit 26e32a365a8ae7d9a5ec525f59326b0616b0ef12 ] Some users have reported a "invalid argument" from cilium-agent wh... — committed to cilium/cilium by jrfastab 5 years ago
cilium: encryption, fix getting started guides create secrects command [ upstream commit 26e32a365a8ae7d9a5ec525f59326b0616b0ef12 ] Some users have reported a "invalid argument" from cilium-agent wh... — committed to cilium/cilium by jrfastab 5 years ago

Most upvoted comments

TLDR: If you have the invalid argument error on v1.5 please try the following command to create the secret instead of the one listed in the getting started guide,

$ kubectl create -n kube-system secret generic cilium-ipsec-keys \
    --from-literal=keys="3 rfc4106(gcm(aes)) $(echo $(dd if=/dev/urandom count=20 bs=1 2> /dev/null| xxd -p -c 64)) 128"

@tushar00jain @kekoav @mythi ^^^

I just spun up an eks instance,

# uname -a
Linux ip-192-168-77-71.us-west-2.compute.internal 4.14.128-112.105.amzn2.x86_64 #1 SMP Wed Jun 19 16:53:40 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

And ran into the above invalid argument errors when using the create-secrets from the getting started guide. In this case the secret generating command I get is the following,

$ kubectl create -n kube-system secret generic cilium-ipsec-keys     --from-literal=keys="3 rfc4106(gcm(aes)) $(echo dd if=/dev/urandom count=20 bs=1 2> /dev/null| xxd -p -c 64) 128"

This is actually wrong! It will give you a secret that is almost correct but has a key of the wrong length and the above invalid argument error happens because the kernel detects the key length is wrong.

The correct command is

$ kubectl create -n kube-system secret generic cilium-ipsec-keys \
    --from-literal=keys="3 rfc4106(gcm(aes)) $(echo $(dd if=/dev/urandom count=20 bs=1 2> /dev/null| xxd -p -c 64)) 128"

The command in the source Documentation directory is the following which also works,

   $ kubectl create -n kube-system secret generic cilium-ipsec-keys \
        --from-literal=keys="3 rfc4106(gcm(aes)) $(echo `dd if=/dev/urandom count=20 bs=1 2> /dev/null| xxd -p -c 64`) 128"

depending on how the shell handles the ` escape around the dd command this will work. But (oops!) if you are following the docs.cilium.io guide the markup dropped the ` and broke the command. Unfortunately when I was testing this I ran the commands from my source version with my editor that did not drop the '.

I’m drafting a patch now. Thanks everyone.

jrfastab on Aug 20, 2019

@tushar00jain I’ll take a look today thanks for the report and sysdump.

jrfastab on Jul 9, 2019