operator-sdk: Operator not getting any requests for related CRD
Bug Report
What did you do?
Followed the tutorial here operator-sdk tutorial
When running locally with
make run ENABLE_WEBHOOKS=false
It seems fine:
go: creating new go.mod: module tmp
go: finding sigs.k8s.io/controller-tools/cmd v0.3.0
go: finding sigs.k8s.io v0.3.0
go: finding sigs.k8s.io/controller-tools/cmd/controller-gen v0.3.0
/Users/yonis/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/Users/yonis/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go run ./main.go
2020-10-18T16:24:20.808+0300 INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
2020-10-18T16:24:20.809+0300 INFO setup starting manager
2020-10-18T16:24:20.809+0300 INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2020-10-18T16:24:20.810+0300 INFO controller Starting EventSource {"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "source": "kind source: /, Kind="}
2020-10-18T16:24:21.715+0300 INFO controller Starting EventSource {"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "source": "kind source: /, Kind="}
2020-10-18T16:24:22.716+0300 INFO controller Starting Controller {"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp"}
2020-10-18T16:24:22.716+0300 INFO controller Starting workers {"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "worker count": 1}
2020-10-18T16:24:33.242+0300 INFO controllers.MgmtApp Creating a new Deployment {"mgmtapp": "default/gc-mgmt-app-test", "Deployment.Namespace": "default", "Deployment.Name": "gc-mgmt-app-test"}
2020-10-18T16:24:36.339+0300 DEBUG controller Successfully Reconciled {"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:24:36.339+0300 DEBUG controller Successfully Reconciled {"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:24:36.470+0300 DEBUG controller Successfully Reconciled {"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:24:36.470+0300 DEBUG controller Successfully Reconciled {"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:25:27.081+0300 INFO controllers.MgmtApp MgmtApp resource not found. Ignoring since object must be deleted {"mgmtapp": "default/gc-mgmt-app-test"}
2020-10-18T16:25:27.081+0300 DEBUG controller Successfully Reconciled {"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:25:27.796+0300 INFO controllers.MgmtApp MgmtApp resource not found. Ignoring since object must be deleted {"mgmtapp": "default/gc-mgmt-app-test"}
2020-10-18T16:25:27.796+0300 DEBUG controller Successfully Reconciled {"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
The deployments and pods are created successfully.
When running remotely on a GKE cluster (the same one the above uses):
2020-10-18T13:26:41.140Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "127.0.0.1:8080"}
2020-10-18T13:26:41.228Z INFO setup starting manager
I1018 13:26:41.233994 1 leaderelection.go:242] attempting to acquire leader lease default/c17466cf....
2020-10-18T13:26:41.234Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
I1018 13:26:58.638603 1 leaderelection.go:252] successfully acquired lease default/c17466cf.
2020-10-18T13:26:58.638Z INFO controller Starting EventSource {"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "source": "kind source: /, Kind="}
2020-10-18T13:26:58.638Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"default","name":"c17466cf.","uid":"733ade09-f2ac-4d17-9168-936d439eed99","apiVersion":"v1","resourceVersion":"2278867"}, "reason": "LeaderElection", "message": "mgmt-app-operator-controller-manager-7d6df57d64-p4rt6_4cbff890-d6e7-4319-8a6d-7e650c313569 became leader"}
2020-10-18T13:26:58.739Z INFO controller Starting EventSource {"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "source": "kind source: /, Kind="}
E1018 13:26:58.743329 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:26:59.786673 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:27:02.213555 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:27:08.019872 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:27:15.795812 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:28:07.215738 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:28:51.286639 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:29:29.780657 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:30:08.576422 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:30:58.953973 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:31:35.513270 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:32:11.611564 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:32:52.440143 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:33:48.317201 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:34:27.113410 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:35:06.029765 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:35:58.609352 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:36:34.809457 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
Obviously the permission issue above seems suspicious
Both are created in the default
namespace
What did you expect to see?
I expect the operator to work on the GKE cluster as it works when being ran locally (against the same GKE cluster)
What did you see instead? Under which circumstances?
The above logs and the deployments/pods are not created
Environment
Operator type:
/language go
Kubernetes cluster type:
GKE
$ operator-sdk version
operator-sdk version: "v1.0.1", commit: "4169b318b578156ed56530f373d328276d040a1b", kubernetes version: "v1.18.2", go version: "go1.15.2 darwin/amd64", GOOS: "darwin", GOARCH: "amd64"
$ go version
(if language is Go)
go version go1.13.15 darwin/amd64
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-gke.1504", GitCommit:"fb367de18820fe57bac5d97ad60b7a2cef8765e9", GitTreeState:"clean", BuildDate:"2020-09-09T01:00:23Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}
Possible Solution
Additional context
The markups added on the Reconsile
function are:
// +kubebuilder:rbac:groups=mgmt-app,resources=mgmtapps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=mgmt-app,resources=mgmtapps/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;
It seems that the operator should be able to list the delpoyment
API
I also did kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user <my-user-name>
As mentioned in the tutorial to make sure my user is privileged
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 18 (10 by maintainers)
Correct.
@zonnie It looks like you’re missing the
watch
permission onpods
.You need your role to include this:
So you would want to update the RBAC marker in your reconciler to look like this:
@camilamacedo86 my
role.yaml
(same one attached above) has these roles as far as I understand:I’m not really sure what versions you are referring to in the
Makefile
- this is myMakefile
:This is my
go.mod
:I’ll try to bump the
controller-runtime
as you suggested anyway