controller-runtime: Cannot Get after Create using Client with Custom Resource

Hi!

I’m tearing my hair out trying to debug this issue.

func testCreation(c client.Client) error {
	config := getTestCuscoServiceConfig()

	err := c.Create(context.Background(), &config)
	if err != nil {
		return errors.Wrap(err, "Could not create CuscoServiceConfig")
	}

	new_csc := v1.CuscoServiceConfig{}
	err = c.Get(context.Background(), client.ObjectKey{Name: "test-config", Namespace: "cusco"}, &new_csc)
	if err != nil {
		return errors.Wrap(err, "Could not get CuscoServiceConfig")
	}

	return nil
}

When run, I get this error:

Could not get CuscoServiceConfig: CuscoServiceConfig.cusco.shopify.io \"test-config\" not found"

Outside the program, though:

$ kubectl get cuscoserviceconfigs test-config -n cusco -o yaml
apiVersion: cusco.shopify.io/v1
kind: CuscoServiceConfig
metadata:
  creationTimestamp: "2019-02-28T15:35:09Z"
  generation: 1
  name: test-config
  namespace: cusco
  resourceVersion: "21614"
  selfLink: /apis/cusco.shopify.io/v1/namespaces/cusco/cuscoserviceconfigs/test-config
  uid: 6e8bb6e3-3b6e-11e9-932e-02423b5580eb
spec:
  init-container:
    enabled: false
    percentage: 75
  istio-version: 1.1.0
  proxy:
    enabled: true
    percentage: 25

Stepping through the code, my theory is that the Create call doesn’t store the object in the cache, and the Get call only looks at the cache.

How should I go about debugging this? Is this because CuscoServiceConfig is a CustomResource?

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 22 (12 by maintainers)

Commits related to this issue

Merge pull request #343 from fanzhangio/fix-domain Fix domain resource issues in controller — committed to kubernetes-sigs/controller-runtime by k8s-ci-robot 6 years ago
Reduce test flakiness * Use a KubeClient in EnvTest that always hits APIServer to avoid cache latency and inconsistency issues that can cause races and intermittent test failures. See https... — committed to matthchr/azure-service-operator by matthchr 3 years ago
Reduce test flakiness * Use a KubeClient in EnvTest that always hits APIServer to avoid cache latency and inconsistency issues that can cause races and intermittent test failures. See https... — committed to matthchr/azure-service-operator by matthchr 3 years ago
Reduce test flakiness * Use a KubeClient in EnvTest that always hits APIServer to avoid cache latency and inconsistency issues that can cause races and intermittent test failures. See https... — committed to matthchr/azure-service-operator by matthchr 3 years ago
Reduce test flakiness * Use a KubeClient in EnvTest that always hits APIServer to avoid cache latency and inconsistency issues that can cause races and intermittent test failures. See https... — committed to matthchr/azure-service-operator by matthchr 3 years ago
Reduce test flakiness * Use a KubeClient in EnvTest that always hits APIServer to avoid cache latency and inconsistency issues that can cause races and intermittent test failures. See https... — committed to matthchr/azure-service-operator by matthchr 3 years ago
Improve test reliability (#1960) * Reduce test flakiness * Use a KubeClient in EnvTest that always hits APIServer to avoid cache latency and inconsistency issues that can cause races and in... — committed to Azure/azure-service-operator by matthchr 3 years ago

Most upvoted comments

ah, yeah, this is probably a bit of a confusing statement, but don’t use the manager client in tests. The manager-provided client is designed to do the right thing for controllers by default (which is to read from caches, meaning that it’s not strongly consistent), which means it probably does the wrong thing for tests (which almost certainly want strong consistency).

The suggested pattern is to construct a new client using client.New for tests (which will provide you with a direct API client that doesn’t have any special behavior).

This is something that we need to document better, but generally, my *_suite_test.go files (the general suite setup for a particular package) look like:

var testenv *envtest.Environment
var cfg *rest.Config
var client client.Client

var _ = BeforeSuite(func() {
    logf.SetLogger(zap.LoggerTo(GinkgoWriter, true))

    testenv = &envtest.Environment{}

    var err error
    cfg, err = testenv.Start()
    Expect(err).NotTo(HaveOccurred())

    client, err = client.New(cfg, client.Options{Scheme: myscheme.Scheme})
    Expect(err).NotTo(HaveOccurred())
})

var _ = AfterSuite(func() {
    testenv.Stop()
})

(this is very similar to a lot of the test suite setup files in controller-runtime itself)

+11

DirectXMan12 on Mar 4, 2019

(also, in case it wasn’t clear, while it’s fine (and generally preferable) to write tests expecting read-after-write consistency, you shouldn’t do that for controllers. Kubernetes favors a kind-of 2-phase approach – do your reads, process, do some writes, and return, letting the requeuing deal with the next cycle of reads if they’re necessary, but don’t try to read an object after you’ve written it).

DirectXMan12 on Mar 4, 2019