ingress-nginx: RBAC ClusterRoleBinding conflicts with itself when 2 separate nginx ingresses are installed
If you install Nginx ingress in a namespace (let’s call it alpha) and then install the same nginx in a separate namespace (we will call beta) under the same name, the ClusterRoleBinding conflicts between the two
Alpha --> Nginx-ingress Beta —> nginx-ingress
If the two nginx-ingress controllers have the same name, but in different namespaces it breaks on installation of the second installation complaining that the clusterrolebinding has already been installed
**NGINX Ingress controller version: Version=1.12.1 GitCommit=6f72db6030daa9afd567fd7faf9d5fffac9c7c8f Date=2021-09-08T13:39:53Z
Kubernetes version (use kubectl version):
Client Version: version.Info{Major:“1”, Minor:“20”, GitVersion:“v1.20.5”, GitCommit:“6b1d87acf3c8253c123756b9e61dac642678305f”, GitTreeState:“clean”, BuildDate:“2021-03-18T01:10:43Z”, GoVersion:“go1.15.8”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“18+”, GitVersion:“v1.18.20-eks-8c579e”, GitCommit:“8c579edfc914f013ff48b2a2b2c1308fdcacc53f”, GitTreeState:“clean”, BuildDate:“2021-07-31T01:34:13Z”, GoVersion:“go1.13.15”, Compiler:“gc”, Platform:“linux/amd64”}
Environment:
-
Cloud provider or hardware configuration: AWK --> EKS 1.18
-
OS (e.g. from /etc/os-release): AWS EKS AMI
-
Kernel (e.g.
uname -a): Unable to log into host node -
Install tools: Helm, EKS
-
Basic cluster related info:
kubectl versionkubectl get nodes -o wide
-
How was the ingress-nginx-controller installed: NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION mongodb test-neweks 1 2021-10-04 22:20:09.60238499 -0400 EDT deployed mongodb-0.1.0 1.0 nginx-ingress test-neweks 1 2021-10-04 22:19:49.90250833 -0400 EDT deployed nginx-ingress-0.10.1 1.12.1
USER-SUPPLIED VALUES: certToken: controller: appprotect: enable: false config: entries: max-fails: “0” worker-connections: “8192” enableCustomResources: false healthStatus: true ingressClass: test-neweks-ingress kind: deployment replicaCount: 1 service: type: ClusterIP watchNamespace: test-neweks prometheus: create: false timestamp: ‘“2021-10-0422:19:45.661497”’
Exact same configuration for second controller, but a different namespace. The second controller does not install due to naming conflict.
-
Current State of the controller: First controller is running, second won’t install
-
Current state of ingress object, if applicable: Ingress object is running fine, but second ingress won’t install due to naming conflict
-
Others:
- Any other related information like ;
- copy/paste of the snippet (if applicable)
kubectl describe ...of any custom configmap(s) created and in use- Any other related information that may help
- Any other related information like ;
What happened:
When giving nginx a specific name installing with Helm, the second install of the same name breaks even when it’s under a different namespace
$ helm install nginx-stable/nginx-ingress --namespace test-neweks --name-template booya NAME: booya LAST DEPLOYED: Tue Oct 5 10:03:42 2021 NAMESPACE: test-neweks STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: The NGINX Ingress Controller has been installed. $ helm install nginx-stable/nginx-ingress --namespace test2-neweks --name-template booya Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole “booya-nginx-ingress” in namespace “” exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key “meta.helm.sh/release-namespace” must equal “test2-neweks”: current value is “test-neweks”
What you expected to happen:
The RBAC cluster-wide resource should have the namespace attached to it’s name so that it doesn’t conflict with other cluster-wide resources How to reproduce it: ##Install Kubernetes from whatever source works for you Create 2 namespaces
Install the ingress controller in one namespace under a pre-defined name
helm install nginx-stable/nginx-ingress --namespace test-neweks --name-template booya
Install the second ingress controller in the other namespace under the same name
helm install nginx-stable/nginx-ingress --namespace test2-neweks --name-template booya
Anything else we need to know:
The RBAC cluster role and specifically the ClusterRoleBinding needs to include the Namespace in the naming convention on the helm chart
/kind bug
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 21 (8 by maintainers)
You’re missing the point of this issue if you think that the name override works for this issue.
If you have the SAME named nginx installation on two different namespaces, it breaks. It should not, this is not expected behaviour.
This is particularly important if your DEV and QA environments live on a single cluster, or if you’re dealing with multi-tenant installations. Ironically NGINX is exactly the tool you need for multitenant operations and that’s what the rest of the package is set up nicely to do!
…except for this RBAC
I think @ax-jgriffiths is experiencing a slightly different problem. I am not sure if ingress-nginx was ever intended to run two ingress-nginx controllers inside the same namespace. I think they would at least have to be in separate namespaces. Perhaps I’m wrong.
I think the fundamental issue here, is that even when deployed into separate namespaces, with
scope: enabled: true, a non-uniquely-named ClusterRole and ClusterRoleBinding are created, and subsequent ingress installs (into unique namespaces) will corrupt each others installation.I am not sure if the ClusterRole* is required if everything is to be scoped to a namespace, if not, I guess we can create a single Role, and then use a RoleBinding per namespace.
If ClusterRole* is required, then we’ll need to create a uniquely named ClusterRoleBinding per namespace.
Yes, @rrichardson that is EXACTLY what is needed. The name on the ClusterRoleBinding needs to have the NAMESPACE included on the name. PLEASE!
It’s literally a one line change that affects almost nothing, but would make a huge difference to many folks out there.
You can actually run 2 of them in the same namespace because they’re forced to have different names, HOWEVER you CANNOT run 2 in different namespaces that both have the SAME name.
The Watch-namespace paramater does NOT change the name of the associated RBAC ClusterRoleBinding
If you look at the file, the ONLY thing that changes that value is the NAME of the package and that’s the issue. Which namespace is being watched doesn’t affect this bug at all.