operator-sdk: Operator not getting any requests for related CRD

Bug Report

What did you do?

Followed the tutorial here operator-sdk tutorial

When running locally with make run ENABLE_WEBHOOKS=false

It seems fine:

go: creating new go.mod: module tmp
go: finding sigs.k8s.io/controller-tools/cmd v0.3.0
go: finding sigs.k8s.io v0.3.0
go: finding sigs.k8s.io/controller-tools/cmd/controller-gen v0.3.0
/Users/yonis/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/Users/yonis/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go run ./main.go
2020-10-18T16:24:20.808+0300	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2020-10-18T16:24:20.809+0300	INFO	setup	starting manager
2020-10-18T16:24:20.809+0300	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2020-10-18T16:24:20.810+0300	INFO	controller	Starting EventSource	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "source": "kind source: /, Kind="}
2020-10-18T16:24:21.715+0300	INFO	controller	Starting EventSource	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "source": "kind source: /, Kind="}
2020-10-18T16:24:22.716+0300	INFO	controller	Starting Controller	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp"}
2020-10-18T16:24:22.716+0300	INFO	controller	Starting workers	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "worker count": 1}
2020-10-18T16:24:33.242+0300	INFO	controllers.MgmtApp	Creating a new Deployment	{"mgmtapp": "default/gc-mgmt-app-test", "Deployment.Namespace": "default", "Deployment.Name": "gc-mgmt-app-test"}
2020-10-18T16:24:36.339+0300	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:24:36.339+0300	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:24:36.470+0300	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:24:36.470+0300	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:25:27.081+0300	INFO	controllers.MgmtApp	MgmtApp resource not found. Ignoring since object must be deleted	{"mgmtapp": "default/gc-mgmt-app-test"}
2020-10-18T16:25:27.081+0300	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:25:27.796+0300	INFO	controllers.MgmtApp	MgmtApp resource not found. Ignoring since object must be deleted	{"mgmtapp": "default/gc-mgmt-app-test"}
2020-10-18T16:25:27.796+0300	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}

The deployments and pods are created successfully.

When running remotely on a GKE cluster (the same one the above uses):

2020-10-18T13:26:41.140Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": "127.0.0.1:8080"}
2020-10-18T13:26:41.228Z	INFO	setup	starting manager
I1018 13:26:41.233994       1 leaderelection.go:242] attempting to acquire leader lease  default/c17466cf....
2020-10-18T13:26:41.234Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
I1018 13:26:58.638603       1 leaderelection.go:252] successfully acquired lease default/c17466cf.
2020-10-18T13:26:58.638Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "source": "kind source: /, Kind="}
2020-10-18T13:26:58.638Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ConfigMap","namespace":"default","name":"c17466cf.","uid":"733ade09-f2ac-4d17-9168-936d439eed99","apiVersion":"v1","resourceVersion":"2278867"}, "reason": "LeaderElection", "message": "mgmt-app-operator-controller-manager-7d6df57d64-p4rt6_4cbff890-d6e7-4319-8a6d-7e650c313569 became leader"}
2020-10-18T13:26:58.739Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "source": "kind source: /, Kind="}
E1018 13:26:58.743329       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:26:59.786673       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:27:02.213555       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:27:08.019872       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:27:15.795812       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:28:07.215738       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:28:51.286639       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:29:29.780657       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:30:08.576422       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:30:58.953973       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:31:35.513270       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:32:11.611564       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:32:52.440143       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:33:48.317201       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:34:27.113410       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:35:06.029765       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:35:58.609352       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:36:34.809457       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope

Obviously the permission issue above seems suspicious

Both are created in the default namespace

What did you expect to see?

I expect the operator to work on the GKE cluster as it works when being ran locally (against the same GKE cluster)

What did you see instead? Under which circumstances?

The above logs and the deployments/pods are not created

Environment

Operator type:

/language go

Kubernetes cluster type:

GKE

$ operator-sdk version

operator-sdk version: "v1.0.1", commit: "4169b318b578156ed56530f373d328276d040a1b", kubernetes version: "v1.18.2", go version: "go1.15.2 darwin/amd64", GOOS: "darwin", GOARCH: "amd64"

$ go version (if language is Go)

go version go1.13.15 darwin/amd64

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-gke.1504", GitCommit:"fb367de18820fe57bac5d97ad60b7a2cef8765e9", GitTreeState:"clean", BuildDate:"2020-09-09T01:00:23Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}

Possible Solution

Additional context

The markups added on the Reconsile function are:

// +kubebuilder:rbac:groups=mgmt-app,resources=mgmtapps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=mgmt-app,resources=mgmtapps/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;

It seems that the operator should be able to list the delpoyment API

I also did kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user <my-user-name> As mentioned in the tutorial to make sure my user is privileged

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 18 (10 by maintainers)

Most upvoted comments

@joelanford So the informer requires a list for the initial population and watch for future changes ?

Correct.

@zonnie It looks like you’re missing the watch permission on pods.

You need your role to include this:

- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - watch

So you would want to update the RBAC marker in your reconciler to look like this:

// +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch;

@camilamacedo86 my role.yaml (same one attached above) has these roles as far as I understand:


---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: manager-role
rules:
- apiGroups:
  - apps
  resources:
  - deployments
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
- apiGroups:
  - mgmt-app.guardicore.com
  resources:
  - mgmtapps
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - mgmt-app.guardicore.com
  resources:
  - mgmtapps/status
  verbs:
  - get
  - patch
  - update

I’m not really sure what versions you are referring to in the Makefile - this is my Makefile:

# Current Operator version
VERSION ?= 0.0.1
# Default bundle image tag
BUNDLE_IMG ?= controller-bundle:$(VERSION)
# Options for 'bundle-build'
ifneq ($(origin CHANNELS), undefined)
BUNDLE_CHANNELS := --channels=$(CHANNELS)
endif
ifneq ($(origin DEFAULT_CHANNEL), undefined)
BUNDLE_DEFAULT_CHANNEL := --default-channel=$(DEFAULT_CHANNEL)
endif
BUNDLE_METADATA_OPTS ?= $(BUNDLE_CHANNELS) $(BUNDLE_DEFAULT_CHANNEL)

# Image URL to use all building/pushing image targets
IMG ?= controller:latest
# Produce CRDs that work back to Kubernetes 1.11 (no version conversion)
CRD_OPTIONS ?= "crd:trivialVersions=true"

# Get the currently used golang install path (in GOPATH/bin, unless GOBIN is set)
ifeq (,$(shell go env GOBIN))
GOBIN=$(shell go env GOPATH)/bin
else
GOBIN=$(shell go env GOBIN)
endif

all: manager

# Run tests
ENVTEST_ASSETS_DIR=$(shell pwd)/testbin
test: generate fmt vet manifests
	mkdir -p ${ENVTEST_ASSETS_DIR}
	test -f ${ENVTEST_ASSETS_DIR}/setup-envtest.sh || curl -sSLo ${ENVTEST_ASSETS_DIR}/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/master/hack/setup-envtest.sh
	source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile cover.out

# Build manager binary
manager: generate fmt vet
	go build -o bin/manager main.go

# Run against the configured Kubernetes cluster in ~/.kube/config
run: generate fmt vet manifests
	go run ./main.go

# Install CRDs into a cluster
install: manifests kustomize
	$(KUSTOMIZE) build config/crd | kubectl apply -f -

# Uninstall CRDs from a cluster
uninstall: manifests kustomize
	$(KUSTOMIZE) build config/crd | kubectl delete -f -

# Deploy controller in the configured Kubernetes cluster in ~/.kube/config
deploy: manifests kustomize
	cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG}
	$(KUSTOMIZE) build config/default | kubectl apply -f -

# Generate manifests e.g. CRD, RBAC etc.
manifests: controller-gen
	$(CONTROLLER_GEN) $(CRD_OPTIONS) rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases

# Run go fmt against code
fmt:
	go fmt ./...

# Run go vet against code
vet:
	go vet ./...

# Generate code
generate: controller-gen
	$(CONTROLLER_GEN) object:headerFile="hack/boilerplate.go.txt" paths="./..."

# Build the docker image
docker-build: test
	docker build . -t ${IMG}

# Push the docker image
docker-push:
	docker push ${IMG}

# find or download controller-gen
# download controller-gen if necessary
controller-gen:
ifeq (, $(shell which controller-gen))
	@{ \
	set -e ;\
	CONTROLLER_GEN_TMP_DIR=$$(mktemp -d) ;\
	cd $$CONTROLLER_GEN_TMP_DIR ;\
	go mod init tmp ;\
	go get sigs.k8s.io/controller-tools/cmd/controller-gen@v0.3.0 ;\
	rm -rf $$CONTROLLER_GEN_TMP_DIR ;\
	}
CONTROLLER_GEN=$(GOBIN)/controller-gen
else
CONTROLLER_GEN=$(shell which controller-gen)
endif

kustomize:
ifeq (, $(shell which kustomize))
	@{ \
	set -e ;\
	KUSTOMIZE_GEN_TMP_DIR=$$(mktemp -d) ;\
	cd $$KUSTOMIZE_GEN_TMP_DIR ;\
	go mod init tmp ;\
	go get sigs.k8s.io/kustomize/kustomize/v3@v3.5.4 ;\
	rm -rf $$KUSTOMIZE_GEN_TMP_DIR ;\
	}
KUSTOMIZE=$(GOBIN)/kustomize
else
KUSTOMIZE=$(shell which kustomize)
endif

# Generate bundle manifests and metadata, then validate generated files.
.PHONY: bundle
bundle: manifests
	operator-sdk generate kustomize manifests -q
	cd config/manager && $(KUSTOMIZE) edit set image controller=$(IMG)
	$(KUSTOMIZE) build config/manifests | operator-sdk generate bundle -q --overwrite --version $(VERSION) $(BUNDLE_METADATA_OPTS)
	operator-sdk bundle validate ./bundle

# Build the bundle image.
.PHONY: bundle-build
bundle-build:
	docker build -f bundle.Dockerfile -t $(BUNDLE_IMG) .

This is my go.mod:

module github.com/guardicore.com/mgmt-app-operator

go 1.13

require (
	github.com/go-logr/logr v0.1.0
	github.com/onsi/ginkgo v1.14.2
	github.com/onsi/gomega v1.10.3
	k8s.io/api v0.18.6
	k8s.io/apimachinery v0.18.6
	k8s.io/client-go v0.18.6
	sigs.k8s.io/controller-runtime v0.6.2
)

I’ll try to bump the controller-runtime as you suggested anyway