rancher: Error related to services/proxy when viewing metrics in Rancher UI

Rancher Cluster:

Rancher version: 2.6.4 Number of nodes: 1

Node OS version: Ubuntu 20.04.3 LTS

Downstream Cluster:

Number of Downstream clusters: 1 Node OS: Ubuntu 20.04.3 LTS RKE/RKE2/K3S version: RKE Kubernetes version: v1.20.4 CNI:

Longhorn:

Longhorn version: CPU per node: Memory per node: Disk type: HDD/SSD/NVMe Network bandwidth between the nodes:

Other:

Underlying Infrastructure: AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal

SURE-4497

Issue description:

When viewing cluster metrics, or metrics for a Workload pod, the following error is shown:

Error
services "http:rancher-monitoring-grafana:80" is forbidden: User "userID" cannot create resource "services/proxy" in API group "" in the namespace "cattle-monitoring-system"

Business impact:

Troubleshooting steps:

Repro steps:

1. Enable monitoring v2 on downstream cluster
2. Create a project in this downstream cluster
3. Create a namespace in this project
4. Create an nginx workload in this namespace
5. Edit the members of the project and add a user with the “Read-only” role
6. Run this in the kubectl shell for the cluster:

   kubectl create clusterrolebinding view-monitoring-ui --clusterrole=monitoring-ui-view --user=userID
   

7. Now this user should be able to login and explore the cluster to view cluster metrics or pod metrics under Workloads

Workaround:

Is workararound available and implemented? Yes What is the workaround: Add the ‘create’ verb to the monitoring-ui-view ClusterRole for the services/proxy resource.

Actual behavior:

During step 7 in the repro steps, the following error is seen when the metrics are viewed:

Error
services "http:rancher-monitoring-grafana:80" is forbidden: User "userID" cannot create resource "services/proxy" in API group "" in the namespace "cattle-monitoring-system"

Expected behavior:

No error should be seen