argo-workflows: Controller: `invalid config map object received in config watcher. Ignored processing` with CPU spike

edits below by agilgur5 for clarity as this issue is very popular

Updates summary

edits above by agilgur5 for clarity as this issue is very popular


Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :latest
  • I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

We’re migrating our Argo workflow from v3.0.2 hosted in an old Rancher cluster to the last version in an AWS EKS cluster.

Ramdomically the workflow controller increases de CPU usage and starts to show this error. level=error msg="invalid config map object received in config watcher. Ignored processing"

If we remove the pod, the new one starts well, but after 1 or 2 days (random), the error reappears.

The last one started yesterday at the moment without workflow running nor scheduled tasks on the infra side.

We don’t have an error in the k8s API log to indicate some EKS outage.

The Argo server component doesn’t have errors.

We don’t have this error in the old version.

Env: Argo: v3.4.10 Installation method: Official Helm Chart (0.33.1) K8s Cluster: AWS EKS 1.27


Workflow-controller k8s config map:

Name:         argo-workflow-controller-configmap
Namespace:    argo
Labels:       app.kubernetes.io/component=workflow-controller
              app.kubernetes.io/instance=argo
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=argo-workflows-cm
              app.kubernetes.io/part-of=argo-workflows
              helm.sh/chart=argo-workflows-0.33.1
Annotations:  meta.helm.sh/release-name: argo
              meta.helm.sh/release-namespace: argo
data:
  metricsConfig:
    enabled: true
    path: /metrics
    port: 9090
    ignoreErrors: false
    secure: false
  persistence:
    archive: true
    archiveTTL: 60d
    postgresql:
      database: argo
      host: <REDACTED>
      passwordSecret:
        key: password
        name: argo-postgres-config
      port: 5432
      ssl: false
      sslMode: disabled
      tableName: argo_workflows
      userNameSecret:
        key: username
        name: argo-postgres-config
  workflowDefaults:
    spec:
      ttlStrategy:
        secondsAfterFailure: 604800
  sso:
    issuer: https://<REDACTED>
    clientId:
      name: argo-server-clientid
      key: clientid
    clientSecret:
      name: argo-server-clientsecret
      key: clientsecret
    redirectUrl: https://<REDACTED>/oauth2/callback
    rbac:
      enabled: true
    scopes: 
      - email
      - profile
      - openid
    issuerAlias: https://<REDACTED>
    sessionExpiry: 240h
  nodeEvents:
    enabled: false

Resource Usage image

Version

v3.4.10

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

N/A Since it starts randomly, we usually don’t have any workflows running

Logs from the workflow controller

Before (nothing):

After (Lines like that appear several times - I only have it in the container output):

time="2023-08-23T09:41:01.370Z" level=error msg="invalid config map object received in config watcher. Ignored processing"

Logs from in your workflow’s wait container

N/A

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 34
  • Comments: 77 (11 by maintainers)

Commits related to this issue

Most upvoted comments

@agilgur5, I’m so sorry for my delay. I had some unforeseen circumstances personal/professional but now I’m back and, if possible, I would like to finish this PR.

Is there a workaround for this issue ? Our logs are being flooded with the particular error.

This issue is also affecting us on: AWS EKS v1.28.4 Argo Worflows 3.5.2

The controller was eating up a lot of CPU and MEM while producing the error messages with a really high rate. This has immediate impact on the cost of logging as it’s flooded by the message in the subject.

We restarted the controller and now it produces the following message every minute: listers.go:79] can not retrieve list of objects using index : Index with name namespace does not exist

@dcerbino would u be able to try the vanilla argo release on at least k8s 1.27.13 or 1.28.9 or 1.29.4 or 1.30.x? the k8s regression fix was backported into these minor versions

do you have any idea on when the fix will be available on the official helm chart?

The official Helm Charts have separate maintainers, so I can’t tell you for them. They generally update pretty quickly and I think have renovate doing automatic dep updates now: https://github.com/argoproj/argo-helm/pull/2651

They also only release patches for the latest minor (as it requires having separate release branches), unless that’s changed recently. So only for 3.5.x right now.

Although you don’t need the Helm Chart to update to update your image version, you can specify it in your values.yaml. #11855 also makes no CRD changes.

I’m currently the release manager for 3.5.x, so as soon as #11855 is merged, I’ll backport it to the release-3.5 branch, which releases every ~2-4 weeks

Do you know what additional review/steps are required to get this into main?

The ones listed in the PR. Since I took over the PR as author, I’m no longer reviewer either.

See also https://justinmayer.com/posts/any-updates/, https://sindresorhus.com/blog/issue-bumping

Hello @agilgur5 do you have any idea on when the fix will be available on the official helm chart?

HI @agilgur5 I can confirm that we are running it in kubernetes 1.29. No significant cpu change. Memory change was about 3-4 Mbytes before 111 Mbyte after 115 Mbyte. Even in this case not a significant change. Memory image CPU image the spike on the cpu where for the bux in the frst fix. and we applied the change with the second fix on April 11

Just want to chime in and say we are also experiencing this issue. Thank you for your work @agilgur5 and @juranir!! I’m not familiar with the internals of Argo but if there’s anything we can do to test things out (using a pre-release, etc) please let me know.

This one bit us ingesting 81TiB of logs into DataDog over a month resuling in some fairly high ingestion charges, we had an index exclusion filter however recently we had a larger spike in ingestion volume of this error across a number of our Argo instances simultaneously resulting in an unexpected ingestion volume increase and while we were saved not indexing these logs we still incurred the ingestion charges for them.

Currently trying to filter these out at the logging agent so we stop sending these logs but a long term fix would be welcomed.

Update: the following log filter applied as a pod annotation appears to work for us to not send the spam to DD by filtering it out at the agent:

ad.datadoghq.com/controller.logs: >-
  [{
    "source": "workflow-controller",
    "service": "workflow-controller",
    "log_processing_rules": [{
      "type": "exclude_at_match",
      "name": "exclude_config_map_errors",
      "pattern" : "invalid config map object received in config watcher. Ignored processing"
    }]
  }]

That might make sense, since there is a continue there currently. I would think that re-creating might result in the same problem though, but idk. I still don’t know why this is happening more in k8s 1.27 specifically or what the root cause is really.

In either case though, we’re looking to replace the RetryWatcher with a proper Informer in #11855 per discussion above as it handles certain edge cases better according to its docs.

We’re running 3.4.9 and have seen this happen twice in the last month, where the controller starts logging multiple times per second the line level=error msg="invalid config map object received in config watcher. Ignored processing" and starts consuming a lot more resources, and the problems do not go away until we delete the pod. The config map itself is properly configured, and after a few days of running ok, it suddenly begins to spam that error log.

For context, the watcher that logs that error is actually watching the wrong namespace when both --namespaced and --managed-namespace=<other-namespace> are used (see https://github.com/argoproj/argo-workflows/issues/11463). I have no idea whether that is related to the nil event or not, but it leads to the watcher actually watching a non-existent config map.

The same issue happened with me as well

FTR, this is still happening for me with latest versions of all components.

After a few hours, the controller starts rapidly logging the error, consuming excessive CPU and network resources until the pod is killed.

Kubernetes: 1.32.4 Argo-Workflows: argoproj/workflow-controller:v3.6.7

@tooptoop4 and @agilgur5 We have just bumped up argo workflow in prod. No error in stage until now. So this bug seems to be fixed with helm chart version 0.41.11 and argo workflow v3.5.8 in K8s Rev: v1.29.4.

Thanks for providing those metrics! Awesome! That’s actually better memory than I anticipated, although it is proportional to number of ConfigMaps so a bit dependent on cluster (the biggest optimization I made was to make the cache only store name and namespace of ConfigMaps where possible, which should only be a few bytes each)

Hello @agilgur5 It seams the the fix is working well. We got no error from the new controller in our stage env.

maybe this is relevant? https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#bug-or-regression-2 mentions:

Kube-apiserver: fixes a 1.27+ regression in watch stability by serving watch requests without a resourceVersion from the watch cache by default, as in <1.27 (disabling the change in #115096 by default). This mitigates the impact of an etcd watch bug (https://github.com/etcd-io/etcd/pull/17555). If the 1.27 change in #115096 to serve these requests from underlying storage is still desired despite the impact on watch stability, it can be re-enabled with a WatchFromStorageWithoutResourceVersion feature gate. (https://github.com/kubernetes/kubernetes/pull/123935, @serathius) [SIG API Machinery and Etcd]

Thanks @agilgur5. I’ve just pulled new image and no error in the log now. Again thanks for your help. I will let you now about the original bug.

I’ll push a patch for that shortly, thanks for testing and providing config info!

pushed up agilgur5/argo-workflow-controller:fix-configmap-informer-v2 for testing

I also overwrote agilgur5/argo-workflow-controller:fix-configmap-informer to the new version

Hello @agilgur5, Thx for your help. I’ve manually rollback to standard helm release. No role error in this case. Then I’ve edited the deployment of argo controller to pull your image and I’ve again logs error. So it seams the the role, SA and binding is ok but for some reason your fix fail to read list the configmap.

I will attach SA role and rolebinding. rolebinding.yaml.txt sa.yaml.txt role.yaml.txt

Thanks for the quick response @dcerbino

However, I believe it’s prudent to first update our Helm chart to version 0.41.1

It’s not strictly necessary. The main parts in the Helm changelog are the past few Argo patch updates. That would make for less differences that could be confounding variables though. But if you’re just verifying the CPU and logs, the other changes shouldn’t have significant impact on either.

configure the controller.image.tag to fix-configmap-informer

The image is uploaded to my personal DockerHub, so more than that: also registry to "" and repository to agilgur5/argo-workflow-controller

I’m also experiencing this issue. I did a major K8s upgrade from v 1.24 to 1.29 on 03/25/24. No issues for almost exactly 7 days. Then at what appears to be very close to the exact minute of 7 days after the upgrade, I started receiving CloudWatch Billing alerts on 04/01/24. After tracking those down, it was clear that the cause was a 100X spike in log volume (CloudWatch measures in bytes).

image

It’s odd that this happened nearly exactly 7 days after the upgrade. Logs were stable for the first 2-3 hours after midnight on 04/01/24 - which indicates that the issue wasn’t caused by a new month/day/week.

@toblich - Did you happen to notice whether the issue was occurring at the 7 day mark for you as well?

#12622 provides a workaround, not yet a full fix. I’ll be taking over #11855 as OP hasn’t responded there in a bit, and that should (hopefully) be the full fix.

It seems this is only happening with kubernetes version 1.27 or later. Does anyone have evidence of it happening on earlier versions?

We are still using client-go from 0.24, which should not be compatible with 1.27, so perhaps that’s the cause here. https://github.com/argoproj/argo-workflows/blob/6c0697928f92160f84e2fc6581551d26e13c0dc5/go.mod#L70

There is some problem with the local environment (I don’t know why). I downloaded the latest version code and made the same change. now the core is working fine without the weird message.

I’ll wait some time and create a PR.

I’m starting to think I’m doing something wrong in the image construction process.

I tried cloning the repo and running the command make workflow-controller-image (without file changes), and the error message listers.go:79] can not retrieve list of objects using index : Index with name namespace does not exist keeps appearing, but if I use the official image it doesn’t.

I tried using Windows and Mac M1(using --platform linux/amd64 parameter) as host.

PS. The build task ends without error.

Not sure about the exact error message, but maybe the resync period duration of 0 from your previous code comment is too short? I think most of the existing informers in the codebase are longer (and also it seems like the resync period parameter is very confusing to informer users from a quick search – it is not the same as a re-list). Also there is a separate issue mentioned above re: managed namespace that potentially impacts it. If you change it to use regular NS and that fixes it, then the error would be due to #11463

I waited a bit to see if the error would appear again. So far this code has solved the initial problem, but a new message has started to appear in the log (every 1min): listers.go:79] can not retrieve list of objects using index : Index with name namespace does not exist

Even with this message, everything still works well, but I believe I made some mistakes that generated this error.

@agilgur5 or does anyone have an idea about that?

PS. Without my change this message will not appear

So in that issue they https://github.com/kubernetes/client-go/issues/334#issuecomment-370561966 using a Reflector or Informer instead to handle several kinds of edge cases including this one. The Argo codebase does use Informers very frequently already, so maybe we should just convert this one. The RetryWatcher source code does mention using Informers instead as well for certain edge cases.

@agilgur5 I’m testing the code below, so far everything is fine, but I’m not sure if it’s the best approach:

    func (wfc *WorkflowController) runConfigMapWatcher() {
    defer runtimeutil.HandleCrash(runtimeutil.PanicHandlers...)
	ctx := context.Background()

	factory := informers.NewSharedInformerFactoryWithOptions(
		wfc.kubeclientset,
		0,
		informers.WithNamespace(wfc.managedNamespace),
	)

	informer := factory.Core().V1().ConfigMaps().Informer()

	stopper := make(chan struct{})

	defer close(stopper)

	informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
		UpdateFunc: func(obj interface{}, newObj interface{}) {
			cm := newObj.(*apiv1.ConfigMap)

			log.Infof("CM Name: %s", cm.GetName())

			if cm.GetName() == wfc.configController.GetName() && wfc.namespace == cm.GetNamespace() {
				log.Infof("Received Workflow Controller config map %s/%s update", cm.Namespace, cm.Name)
				wfc.UpdateConfig(ctx)
			}
			wfc.notifySemaphoreConfigUpdate(cm)
		},
	})
	go informer.Run(stopper)
	if !cache.WaitForCacheSync(stopper, informer.HasSynced) {
		log.Error(fmt.Errorf("Timed out waiting for caches to sync"))
		return
	}
	<-stopper
	}

Maybe this issue helps explain this behavior (I also found other complaints similar to this for projects other than Argo), if so I think we would add a conditional check before casting the object:

   // [...]
   select {
   	case event := <-retryWatcher.ResultChan():
   		if event.Object == nil {
   			return
   		}
   		cm, ok := event.Object.(*apiv1.ConfigMap)
   // [...]

I’m assuming with this change the defer will be executed and a new instance of the watcher will be executed.

Does it make sense to you @agilgur5 ?

I updated the code to try to register the “invalid object” that is causing the error. What is happening is that the event variable is null. I don’t have another error log. I’m using AWS EKS, so I also checked its log and I have no errors.

So I have no idea about this 😦