kubernetes: [GC] controller manager gets error "unable to get REST mapping for kind" for ownerRefs to TPR and add-on APIs
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):
What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):
TPR. GC. OwnerReference. REST mapping.
Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT.
Kubernetes version (use kubectl version):
master. v1.5.1
Environment:
- Cloud provider or hardware configuration: Repro in “hack/local-up-cluster.sh”. But I have seen it in many different env.
- OS (e.g. from /etc/os-release):
- Kernel (e.g.
uname -a): - Install tools:
- Others:
What happened: The controller manager is unresponsive and overloaded with error message:
E0112 18:26:11.690849 21374 garbagecollector.go:594] Error syncing item
&garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:v1.OwnerReference{APIVersion:"etcd.coreos.com/v1beta1", Kind:"Cluster", Name:"example-etcd-cluster", UID:"89603932-d8f4-11e6-94f3-42010af00002", Controller:(*bool)(0xc422850af8)}, Namespace:"default"}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, dependents:map[*garbagecollector.node]struct {}{(*garbagecollector.node)(0xc42075a6c0):struct {}{}, (*garbagecollector.node)(0xc421a0abd0):struct {}{}, (*garbagecollector.node)(0xc421c074d0):struct {}{}}, owners:[]v1.OwnerReference(nil)}:
unable to get REST mapping for kind: Cluster, version: etcd.coreos.com/v1beta1
This error just keeps bloating up without any backoff. This seems that REST mapping didn’t recognize the kind.
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Generic way:
- Create a TPR.
- Create a TPR object “A”
- Create pods that has OwnerReference pointing to “A”.
Easy way:
- Use https://github.com/coreos/etcd-operator
- kubectl create -f example/deployment.yaml
- kubectl create -f example/example-etcd-cluster.yaml
Anything else do we need to know:
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 3
- Comments: 45 (40 by maintainers)
Commits related to this issue
- Make HandleError prevent hot-loops Add an error "handler" that just sleeps for a bit if errors happen more often than 500ms. Manually tested against #39816. — committed to lavalamp/kubernetes by lavalamp 7 years ago
- Make HandleError prevent hot-loops Add an error "handler" that just sleeps for a bit if errors happen more often than 500ms. Manually tested against #39816. — committed to lavalamp/kubernetes by lavalamp 7 years ago
- Make HandleError prevent hot-loops Add an error "handler" that just sleeps for a bit if errors happen more often than 500ms. Manually tested against #39816. — committed to lavalamp/kubernetes by lavalamp 7 years ago
- Make HandleError prevent hot-loops Add an error "handler" that just sleeps for a bit if errors happen more often than 500ms. Manually tested against #39816. — committed to lavalamp/kubernetes by lavalamp 7 years ago
- Merge pull request #40497 from lavalamp/log2much Automatic merge from submit-queue Make HandleError prevent hot-loops Add an error "handler" that just sleeps for a bit if errors happen more often ... — committed to kubernetes/kubernetes by deleted user 7 years ago
- Make HandleError prevent hot-loops Add an error "handler" that just sleeps for a bit if errors happen more often than 500ms. Manually tested against #39816. — committed to hpcloud/kubernetes by lavalamp 7 years ago
- Disable references for now https://github.com/kubernetes/kubernetes/issues/39816 — committed to atlassian/smith by ash2k 7 years ago
- Merge pull request #42862 from caesarxuchao/sync-warning Automatic merge from submit-queue (batch tested with PRs 38805, 42362, 42862) Let GC print specific message for RESTMapping failure Make the... — committed to kubernetes/kubernetes by deleted user 7 years ago
@lavalamp shall we reopen? i do not think this is fixed.
I just installed with kubeadm two days ago. I believe is 1.5.2. I then tried using etcd-operator and it killed my master with out of disk space writing 300MB/s of log from GC