rancher: Dashboard generates too many requests when listing resources

What kind of request is this (question/bug/enhancement/feature request): bug

Steps to reproduce (least amount of steps as possible): 1, Go to cluster explorer 2. Go to ingress page

Result: Depending on the browser, it either completely hangs or works really slowly. It seems to go into infinite loop trying to fetch cluster’s resoruces for some reason.

obraz

It did work correctly on “stable” branch which I believe is 2.4.x

Other details that may be helpful: v2.5.3

Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): v2.5.3
Installation option (single install/HA): HA, set up using RKE

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): local/imported, but created using RKE
Machine type (cloud/VM/metal) and specifications (CPU/memory): metal
Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"archive", BuildDate:"2020-11-25T13:19:56Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Docker version (use docker version):

Client: Docker Engine - Community
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 17:02:52 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:01:20 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

gz#15108

gz#17198

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 6
Comments: 25 (16 by maintainers)

Most upvoted comments

What kind of request is this (question/bug/enhancement/feature request):

Bug

Steps to reproduce (least amount of steps as possible):

Provision a Rancher v2.5 instance with a custom downstream cluster (I used https://github.com/superseb/tf-do-rancher2).
View the (empty) Ingresses view in the Cluster Explorer UI for the custom cluster, observe page loads as expected
Create some ingresses in the cluster:

for i in {1..100}; do
  cat << EOF | kubectl --kubeconfig=kubeconfig.yml apply -f -
  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    name: test-$i
    namespace: default
  spec:
    rules:
    - host: test-2.example.cm
      http:
        paths:
        - backend:
            serviceName: kubernetes
            servicePort: 444
EOF
done

View the Ingresses view in the Cluster Explorer UI for the custom cluster again, now with Ingress resources present

Result:

Observe the page is slow to load and 100s of requests are generated to the Rancher proxied v1 endpoints for apps.relicasets, replicationcontrollers, batch.jobs, apps.daemonsets, batch.cronjobs, apps.deployments, apps.statefulsets (e.g. https://128.199.41.199/k8s/clusters/c-v5kqf/v1/batch.cronjobs).

Other details that may be helpful:

In the observed user environment, the user had c. 1500 deployments and c. 2000 replicasets. The traffic of the 100s of requests for the workload endpoints DoS’s the cluster-agent/the controlplane in the downstream, ballooning cluster-agent memory until it is OOM killed, and the host temporarily freezes

Environment information

Reproduced with Rancher v2.5.5 / v1.17.16-rancher1-1, matching user’s initial environment, and Rancher v2.5.7 / v1.20.5-rancher1-1 to test latest

axeal on Apr 14, 2021