kops: Cluster Nodes & Masters Failing kube-bench Security Checks - Question
I just ran kube-bench (https://github.com/aquasecurity/kube-bench) across my KOPS 1.10 provisioned clusters (Kubernetes 1.10.11 running kope.io/k8s-1.10-debian-jessie-amd64-hvm-ebs-2018-08-17) and I was quite shocked to see a large number of tests failed for both nodes and masters.
I expected a few, but not quite this many.
Is there any reason why there seems to be so many security issues with both node and master machines provisioned with KOPS?
Some output
Workers
[INFO] 2 Worker Node Security Configuration
[INFO] 2.1 Kubelet
[FAIL] 2.1.1 Ensure that the --allow-privileged argument is set to false (Scored)
[FAIL] 2.1.2 Ensure that the --anonymous-auth argument is set to false (Scored)
[FAIL] 2.1.3 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Scored)
[FAIL] 2.1.4 Ensure that the --client-ca-file argument is set as appropriate (Scored)
[FAIL] 2.1.5 Ensure that the --read-only-port argument is set to 0 (Scored)
[FAIL] 2.1.6 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Scored)
[FAIL] 2.1.7 Ensure that the --protect-kernel-defaults argument is set to true (Scored)
[PASS] 2.1.8 Ensure that the --make-iptables-util-chains argument is set to true (Scored)
[FAIL] 2.1.9 Ensure that the --hostname-override argument is not set (Scored)
[FAIL] 2.1.10 Ensure that the --event-qps argument is set to 0 (Scored)
[FAIL] 2.1.11 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored)
[PASS] 2.1.12 Ensure that the --cadvisor-port argument is set to 0 (Scored)
[FAIL] 2.1.13 Ensure that the --rotate-certificates argument is not set to false (Scored)
[FAIL] 2.1.14 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)
[FAIL] 2.1.15 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Not Scored)
[INFO] 2.2 Configuration Files
[FAIL] 2.2.1 Ensure that the kubelet.conf file permissions are set to 644 or more restrictive (Scored)
[FAIL] 2.2.2 Ensure that the kubelet.conf file ownership is set to root:root (Scored)
[FAIL] 2.2.3 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Scored)
[FAIL] 2.2.4 Ensure that the kubelet service file ownership is set to root:root (Scored)
[FAIL] 2.2.5 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored)
[FAIL] 2.2.6 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored)
[WARN] 2.2.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Scored)
[WARN] 2.2.8 Ensure that the client certificate authorities file ownership is set to root:root (Scored)
[FAIL] 2.2.9 Ensure that the kubelet configuration file ownership is set to root:root (Scored)
[FAIL] 2.2.10 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored)
Masters
[INFO] 1 Master Node Security Configuration
[INFO] 1.1 API Server
[PASS] 1.1.1 Ensure that the --anonymous-auth argument is set to false (Scored)
[FAIL] 1.1.2 Ensure that the --basic-auth-file argument is not set (Scored)
[PASS] 1.1.3 Ensure that the --insecure-allow-any-token argument is not set (Scored)
[PASS] 1.1.4 Ensure that the --kubelet-https argument is set to true (Scored)
[FAIL] 1.1.5 Ensure that the --insecure-bind-address argument is not set (Scored)
[FAIL] 1.1.6 Ensure that the --insecure-port argument is set to 0 (Scored)
[PASS] 1.1.7 Ensure that the --secure-port argument is not set to 0 (Scored)
[FAIL] 1.1.8 Ensure that the --profiling argument is set to false (Scored)
[FAIL] 1.1.9 Ensure that the --repair-malformed-updates argument is set to false (Scored)
[PASS] 1.1.10 Ensure that the admission control plugin AlwaysAdmit is not set (Scored)
[FAIL] 1.1.11 Ensure that the admission control plugin AlwaysPullImages is set (Scored)
[FAIL] 1.1.12 Ensure that the admission control plugin DenyEscalatingExec is set (Scored)
[FAIL] 1.1.13 Ensure that the admission control plugin SecurityContextDeny is set (Scored)
[FAIL] 1.1.14 Ensure that the admission control plugin NamespaceLifecycle is set (Scored)
[FAIL] 1.1.15 Ensure that the --audit-log-path argument is set as appropriate (Scored)
[FAIL] 1.1.16 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Scored)
[FAIL] 1.1.17 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Scored)
[FAIL] 1.1.18 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Scored)
[PASS] 1.1.19 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Scored)
[FAIL] 1.1.20 Ensure that the --token-auth-file parameter is not set (Scored)
[FAIL] 1.1.21 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Scored)
[FAIL] 1.1.22 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Scored)
[FAIL] 1.1.23 Ensure that the --service-account-lookup argument is set to true (Scored)
[FAIL] 1.1.24 Ensure that the admission control plugin PodSecurityPolicy is set (Scored)
[FAIL] 1.1.25 Ensure that the --service-account-key-file argument is set as appropriate (Scored)
[FAIL] 1.1.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Scored)
[PASS] 1.1.27 Ensure that the admission control plugin ServiceAccount is set(Scored)
[PASS] 1.1.28 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored)
[PASS] 1.1.29 Ensure that the --client-ca-file argument is set as appropriate (Scored)
[FAIL] 1.1.30 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Not Scored)
[FAIL] 1.1.31 Ensure that the --etcd-cafile argument is set as appropriate (Scored)
[FAIL] 1.1.32 Ensure that the --authorization-mode argument is set to Node (Scored)
[PASS] 1.1.33 Ensure that the admission control plugin NodeRestriction is set (Scored)
[FAIL] 1.1.34 Ensure that the --experimental-encryption-provider-config argument is set as appropriate (Scored)
[WARN] 1.1.35 Ensure that the encryption provider is set to aescbc (Scored)
[FAIL] 1.1.36 Ensure that the admission control plugin EventRateLimit is set (Scored)
[PASS] 1.1.37 Ensure that the AdvancedAuditing argument is not set to false (Scored)
[PASS] 1.1.38 Ensure that the --request-timeout argument is set as appropriate (Scored)
[FAIL] 1.1.39 Ensure that the API Server only makes use of Strong Cryptographic Ciphers ( Not Scored)
[INFO] 1.2 Scheduler
[FAIL] 1.2.1 Ensure that the --profiling argument is set to false (Scored)
[PASS] 1.2.2 Ensure that the --address argument is set to 127.0.0.1 (Scored)
[INFO] 1.3 Controller Manager
[FAIL] 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Scored)
[FAIL] 1.3.2 Ensure that the --profiling argument is set to false (Scored)
[PASS] 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Scored)
[PASS] 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Scored)
[PASS] 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Scored)
[FAIL] 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)
[PASS] 1.3.7 Ensure that the --address argument is set to 127.0.0.1 (Scored)
[INFO] 1.4 Configuration Files
[FAIL] 1.4.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Scored)
[FAIL] 1.4.2 Ensure that the API server pod specification file ownership is set to root:root (Scored)
[FAIL] 1.4.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Scored)
[FAIL] 1.4.4 Ensure that the controller manager pod specification file ownership is set to root:root (Scored)
[FAIL] 1.4.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Scored)
[FAIL] 1.4.6 Ensure that the scheduler pod specification file ownership is set to root:root (Scored)
[FAIL] 1.4.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Scored)
[FAIL] 1.4.8 Ensure that the etcd pod specification file ownership is set to root:root (Scored)
[WARN] 1.4.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Not Scored)
[WARN] 1.4.10 Ensure that the Container Network Interface file ownership is set to root:root (Not Scored)
[FAIL] 1.4.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Scored)
[FAIL] 1.4.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Scored)
[FAIL] 1.4.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Scored)
[FAIL] 1.4.14 Ensure that the admin.conf file ownership is set to root:root (Scored)
[FAIL] 1.4.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Scored)
[FAIL] 1.4.16 Ensure that the scheduler.conf file ownership is set to root:root (Scored)
[FAIL] 1.4.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Scored)
[FAIL] 1.4.18 Ensure that the controller-manager.conf file ownership is set to root:root (Scored)
[INFO] 1.5 etcd
[FAIL] 1.5.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Scored)
[FAIL] 1.5.2 Ensure that the --client-cert-auth argument is set to true (Scored)
[PASS] 1.5.3 Ensure that the --auto-tls argument is not set to true (Scored)
[WARN] 1.5.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Scored)
[WARN] 1.5.5 Ensure that the --peer-client-cert-auth argument is set to true (Scored)
[WARN] 1.5.6 Ensure that the --peer-auto-tls argument is not set to true (Scored)
[WARN] 1.5.7 Ensure that a unique Certificate Authority is used for etcd (Not Scored)
[INFO] 1.6 General Security Primitives
[WARN] 1.6.1 Ensure that the cluster-admin role is only used where required (Not Scored)
[WARN] 1.6.2 Create administrative boundaries between resources using namespaces (Not Scored)
[WARN] 1.6.3 Create network segmentation using Network Policies (Not Scored)
[WARN] 1.6.4 Ensure that the seccomp profile is set to docker/default in your pod definitions (Not Scored)
[WARN] 1.6.5 Apply Security Context to Your Pods and Containers (Not Scored)
[WARN] 1.6.6 Configure Image Provenance using ImagePolicyWebhook admission controller (Not Scored)
[WARN] 1.6.7 Configure Network policies as appropriate (Not Scored)
[WARN] 1.6.8 Place compensating controls in the form of PSP and RBAC for privileged containers usage (Not Scored)
[INFO] 1.7 PodSecurityPolicies
[WARN] 1.7.1 Do not admit privileged containers (Not Scored)
[WARN] 1.7.2 Do not admit containers wishing to share the host process ID namespace (Not Scored)
[WARN] 1.7.3 Do not admit containers wishing to share the host IPC namespace (Not Scored)
[WARN] 1.7.4 Do not admit containers wishing to share the host network namespace (Not Scored)
[WARN] 1.7.5 Do not admit containers with allowPrivilegeEscalation (Not Scored)
[WARN] 1.7.6 Do not admit root containers (Not Scored)
[WARN] 1.7.7 Do not admit containers with dangerous capabilities (Not Scored)
UPDATED WITH FULL LIST AFTER VERSION 1.10.11 UPGRADE
About this issue
- Original URL
- State: open
- Created 6 years ago
- Reactions: 11
- Comments: 22 (8 by maintainers)
Thanks for the list - it is much easier to come up with a checklist than it is to actually implement them in the real world when people have running clusters š We have to think about whether each one of these options breaks people.
That said thereās some good stuff here, like
--anonymous-auth=false. kops passes that one, but my understanding is that most of the other configurations out there would fail that one. So we got lucky on the recent CVE there, but thereās enough in here that we might not be so lucky next time.I do think we should use this list to drive a goal of being the most secure configuration available - I have it on a discussion item for āroadmapā for todayās kops office hours.
As a taster of the complexities though, I spotted
AlwaysPullImages should be set- thatās really difficult. As I understand it, the problem is that kubernetes doesnāt keep track of whether an image was previously pulled using a secret, and so if another tenant has pulled an image with a secret, another user on the same node will be able to pull it. AlwaysPullImages forces everyone to pull, all the time. The problem is that this creates a harder dependency on the image registry being available. Some people might want to be able to run a pod even if their image registry is unavailable, if the image happens to be cached (and k8s does try to schedule pods to nodes where the image is cached). And most people likely arenāt running hostile tenants on the same cluster. So ā¦ itās complicated šLetās tag for 1.13 as a target list (we can pull things into earlier releases as well). (And I think some of these are actually getting fixed in kops 1.11)
FWIW if youāre looking for things to fix out of this list, as default settings, Iād thoroughly recommend looking at these two on the kubelet.
as a priority. This combination allows anyone who can hit the kubelet port (10250/TCP) at a network level, to easily get root access on the underlying node.
Also worth noting that there are newer versions of the benchmark out now which drop some of the recommendations which could cause issues with cluster operation (e.g. the
--allow-privileged=falseone has gone, in favour of PSPs)I did a default kops install today and a couple of other ones you might want to look at changing
--insecure-bind-addressand--insecure-port. Even just on localhost the insecure API port can be very dangerous. It lets anyone who can get access to the localhost on the master node (which could be through a variety of routes) get full cluster admin access easily.Using basic and token auth is generally considered a bad idea as they both store credentials in-clear on disk.
etcd without authentication. Again although this is only bound to localhost, from a security standpoint, unauthenticated etcd is dangerous as it contains all the secrets from the cluster.
A more minor point , but one worth considering. All the keys stored in /srv/kubernetes are valid for 10 years. An attacker who compromises one of those keys gets persistent access to the targeted cluster, as they donāt expire for a long time.