argo-workflows: Controller: `invalid config map object received in config watcher. Ignored processing` with CPU spike
edits below by agilgur5 for clarity as this issue is very popular
Updates summary
- this seems to be caused by upgrading your cluster to k8s 1.27
- an upstream k8s fix was applied in 1.27.13+, 1.28.9+, 1.29.4+, and 1.30+ and confirmed working
- there is a workaround available in Argo 3.5.5, setting
WATCH_CONTROLLER_SEMAPHORE_CONFIGMAPS=false. See #12622- you can also workaround the log spam with log filters, such as this one for DataDog
- a larger fix is under way in #11855
- you can help test this!
edits above by agilgur5 for clarity as this issue is very popular
Pre-requisites
- I have double-checked my configuration
- I can confirm the issues exists when I tested with
:latest - I’d like to contribute the fix myself (see contributing guide)
What happened/what you expected to happen?
We’re migrating our Argo workflow from v3.0.2 hosted in an old Rancher cluster to the last version in an AWS EKS cluster.
Ramdomically the workflow controller increases de CPU usage and starts to show this error.
level=error msg="invalid config map object received in config watcher. Ignored processing"
If we remove the pod, the new one starts well, but after 1 or 2 days (random), the error reappears.
The last one started yesterday at the moment without workflow running nor scheduled tasks on the infra side.
We don’t have an error in the k8s API log to indicate some EKS outage.
The Argo server component doesn’t have errors.
We don’t have this error in the old version.
Env: Argo: v3.4.10 Installation method: Official Helm Chart (0.33.1) K8s Cluster: AWS EKS 1.27
Workflow-controller k8s config map:
Name: argo-workflow-controller-configmap
Namespace: argo
Labels: app.kubernetes.io/component=workflow-controller
app.kubernetes.io/instance=argo
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=argo-workflows-cm
app.kubernetes.io/part-of=argo-workflows
helm.sh/chart=argo-workflows-0.33.1
Annotations: meta.helm.sh/release-name: argo
meta.helm.sh/release-namespace: argo
data:
metricsConfig:
enabled: true
path: /metrics
port: 9090
ignoreErrors: false
secure: false
persistence:
archive: true
archiveTTL: 60d
postgresql:
database: argo
host: <REDACTED>
passwordSecret:
key: password
name: argo-postgres-config
port: 5432
ssl: false
sslMode: disabled
tableName: argo_workflows
userNameSecret:
key: username
name: argo-postgres-config
workflowDefaults:
spec:
ttlStrategy:
secondsAfterFailure: 604800
sso:
issuer: https://<REDACTED>
clientId:
name: argo-server-clientid
key: clientid
clientSecret:
name: argo-server-clientsecret
key: clientsecret
redirectUrl: https://<REDACTED>/oauth2/callback
rbac:
enabled: true
scopes:
- email
- profile
- openid
issuerAlias: https://<REDACTED>
sessionExpiry: 240h
nodeEvents:
enabled: false
Resource Usage
Version
v3.4.10
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.
N/A Since it starts randomly, we usually don’t have any workflows running
Logs from the workflow controller
Before (nothing):
After (Lines like that appear several times - I only have it in the container output):
time="2023-08-23T09:41:01.370Z" level=error msg="invalid config map object received in config watcher. Ignored processing"
Logs from in your workflow’s wait container
N/A
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 34
- Comments: 77 (11 by maintainers)
Commits related to this issue
- fix: Refactor the func runConfigMapWatcher to use Informers. Fixes #11657 Signed-off-by: juranir <juranir.santos@gmail.com> — committed to juranir/argo-workflows by juranir 3 years ago
- fix: Refactor newConfigMapInformer to handle ctrl cm. Fixes #11657 Signed-off-by: juranir <juranir.santos@gmail.com> — committed to juranir/argo-workflows by juranir 3 years ago
- feat(controller): option to not watch configmap Fixes https://github.com/argoproj/argo-workflows/issues/11657 Without setting this env variable argo-workflows is unusable with k8s >= 1.27 as logs ... — committed to tooptoop4/argo-workflows by tooptoop4 3 years ago
- fix: temporary stop sending Argo controller logs TDE-1016 (#380) #### Motivation The Argo Controller pod are having this issue https://github.com/argoproj/argo-workflows/issues/11657. This issue is ... — committed to linz/topo-workflows by paulfouquet 3 years ago
- fix: use WATCH_CONFIGMAPS workaround TDE-1015 (#498) #### Motivation In order to reduce the Argo controller logs due to [this issue](https://github.com/argoproj/argo-workflows/issues/11657), we can ... — committed to linz/topo-workflows by paulfouquet 3 years ago
- fix: Refactor the func newConfigMapInformer to watch both main and mgmt ns. Fixes #11657 Signed-off-by: Juranir Santos <juranir.santos@gmail.com> — committed to juranir/argo-workflows by juranir 3 years ago
@agilgur5, I’m so sorry for my delay. I had some unforeseen circumstances personal/professional but now I’m back and, if possible, I would like to finish this PR.
Is there a workaround for this issue ? Our logs are being flooded with the particular error.
This issue is also affecting us on: AWS EKS v1.28.4 Argo Worflows 3.5.2
The controller was eating up a lot of CPU and MEM while producing the error messages with a really high rate. This has immediate impact on the cost of logging as it’s flooded by the message in the subject.
We restarted the controller and now it produces the following message every minute:
listers.go:79] can not retrieve list of objects using index : Index with name namespace does not exist@dcerbino would u be able to try the vanilla argo release on at least k8s 1.27.13 or 1.28.9 or 1.29.4 or 1.30.x? the k8s regression fix was backported into these minor versions
The official Helm Charts have separate maintainers, so I can’t tell you for them. They generally update pretty quickly and I think have
renovatedoing automatic dep updates now: https://github.com/argoproj/argo-helm/pull/2651They also only release patches for the latest minor (as it requires having separate release branches), unless that’s changed recently. So only for 3.5.x right now.
Although you don’t need the Helm Chart to update to update your image version, you can specify it in your
values.yaml. #11855 also makes no CRD changes.I’m currently the release manager for 3.5.x, so as soon as #11855 is merged, I’ll backport it to the
release-3.5branch, which releases every ~2-4 weeksThe ones listed in the PR. Since I took over the PR as author, I’m no longer reviewer either.
See also https://justinmayer.com/posts/any-updates/, https://sindresorhus.com/blog/issue-bumping
Hello @agilgur5 do you have any idea on when the fix will be available on the official helm chart?
HI @agilgur5 I can confirm that we are running it in kubernetes 1.29. No significant cpu change. Memory change was about 3-4 Mbytes before 111 Mbyte after 115 Mbyte. Even in this case not a significant change. Memory
CPU
the spike on the cpu where for the bux in the frst fix.
and we applied the change with the second fix on April 11
Just want to chime in and say we are also experiencing this issue. Thank you for your work @agilgur5 and @juranir!! I’m not familiar with the internals of Argo but if there’s anything we can do to test things out (using a pre-release, etc) please let me know.
This one bit us ingesting 81TiB of logs into DataDog over a month resuling in some fairly high ingestion charges, we had an index exclusion filter however recently we had a larger spike in ingestion volume of this error across a number of our Argo instances simultaneously resulting in an unexpected ingestion volume increase and while we were saved not indexing these logs we still incurred the ingestion charges for them.
Currently trying to filter these out at the logging agent so we stop sending these logs but a long term fix would be welcomed.
Update: the following log filter applied as a pod annotation appears to work for us to not send the spam to DD by filtering it out at the agent:
That might make sense, since there is a
continuethere currently. I would think that re-creating might result in the same problem though, but idk. I still don’t know why this is happening more in k8s 1.27 specifically or what the root cause is really.In either case though, we’re looking to replace the
RetryWatcherwith a properInformerin #11855 per discussion above as it handles certain edge cases better according to its docs.We’re running 3.4.9 and have seen this happen twice in the last month, where the controller starts logging multiple times per second the line
level=error msg="invalid config map object received in config watcher. Ignored processing"and starts consuming a lot more resources, and the problems do not go away until we delete the pod. The config map itself is properly configured, and after a few days of running ok, it suddenly begins to spam that error log.For context, the watcher that logs that error is actually watching the wrong namespace when both
--namespacedand--managed-namespace=<other-namespace>are used (see https://github.com/argoproj/argo-workflows/issues/11463). I have no idea whether that is related to the nil event or not, but it leads to the watcher actually watching a non-existent config map.The same issue happened with me as well
FTR, this is still happening for me with latest versions of all components.
After a few hours, the controller starts rapidly logging the error, consuming excessive CPU and network resources until the pod is killed.
Kubernetes: 1.32.4 Argo-Workflows: argoproj/workflow-controller:v3.6.7
@tooptoop4 and @agilgur5 We have just bumped up argo workflow in prod. No error in stage until now. So this bug seems to be fixed with helm chart version 0.41.11 and argo workflow v3.5.8 in K8s Rev: v1.29.4.
Thanks for providing those metrics! Awesome! That’s actually better memory than I anticipated, although it is proportional to number of ConfigMaps so a bit dependent on cluster (the biggest optimization I made was to make the cache only store
nameandnamespaceof ConfigMaps where possible, which should only be a few bytes each)Hello @agilgur5 It seams the the fix is working well. We got no error from the new controller in our stage env.
maybe this is relevant? https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#bug-or-regression-2 mentions:
Thanks @agilgur5. I’ve just pulled new image and no error in the log now. Again thanks for your help. I will let you now about the original bug.
pushed up
agilgur5/argo-workflow-controller:fix-configmap-informer-v2for testingI also overwrote
agilgur5/argo-workflow-controller:fix-configmap-informerto the new versionHello @agilgur5, Thx for your help. I’ve manually rollback to standard helm release. No role error in this case. Then I’ve edited the deployment of argo controller to pull your image and I’ve again logs error. So it seams the the role, SA and binding is ok but for some reason your fix fail to read list the configmap.
I will attach SA role and rolebinding. rolebinding.yaml.txt sa.yaml.txt role.yaml.txt
Thanks for the quick response @dcerbino
It’s not strictly necessary. The main parts in the Helm changelog are the past few Argo patch updates. That would make for less differences that could be confounding variables though. But if you’re just verifying the CPU and logs, the other changes shouldn’t have significant impact on either.
The image is uploaded to my personal DockerHub, so more than that: also
registryto""andrepositorytoagilgur5/argo-workflow-controllerI’m also experiencing this issue. I did a major K8s upgrade from v 1.24 to 1.29 on 03/25/24. No issues for almost exactly 7 days. Then at what appears to be very close to the exact minute of 7 days after the upgrade, I started receiving CloudWatch Billing alerts on 04/01/24. After tracking those down, it was clear that the cause was a 100X spike in log volume (CloudWatch measures in bytes).
It’s odd that this happened nearly exactly 7 days after the upgrade. Logs were stable for the first 2-3 hours after midnight on 04/01/24 - which indicates that the issue wasn’t caused by a new month/day/week.
@toblich - Did you happen to notice whether the issue was occurring at the 7 day mark for you as well?
#12622 provides a workaround, not yet a full fix. I’ll be taking over #11855 as OP hasn’t responded there in a bit, and that should (hopefully) be the full fix.
maybe caused by https://github.com/kubernetes/kubernetes/issues/121688#issuecomment-1789860849
It seems this is only happening with kubernetes version 1.27 or later. Does anyone have evidence of it happening on earlier versions?
We are still using client-go from 0.24, which should not be compatible with 1.27, so perhaps that’s the cause here. https://github.com/argoproj/argo-workflows/blob/6c0697928f92160f84e2fc6581551d26e13c0dc5/go.mod#L70
There is some problem with the local environment (I don’t know why). I downloaded the latest version code and made the same change. now the core is working fine without the weird message.
I’ll wait some time and create a PR.
I’m starting to think I’m doing something wrong in the image construction process.
I tried cloning the repo and running the command
make workflow-controller-image(without file changes), and the error messagelisters.go:79] can not retrieve list of objects using index : Index with name namespace does not existkeeps appearing, but if I use the official image it doesn’t.I tried using Windows and Mac M1(using --platform linux/amd64 parameter) as host.
PS. The build task ends without error.
Not sure about the exact error message, but maybe the resync period duration of
0from your previous code comment is too short? I think most of the existing informers in the codebase are longer (and also it seems like the resync period parameter is very confusing to informer users from a quick search – it is not the same as a re-list). Also there is a separate issue mentioned above re: managed namespace that potentially impacts it. If you change it to use regular NS and that fixes it, then the error would be due to #11463I waited a bit to see if the error would appear again. So far this code has solved the initial problem, but a new message has started to appear in the log (every 1min):
listers.go:79] can not retrieve list of objects using index : Index with name namespace does not existEven with this message, everything still works well, but I believe I made some mistakes that generated this error.
@agilgur5 or does anyone have an idea about that?
PS. Without my change this message will not appear
@agilgur5 I’m testing the code below, so far everything is fine, but I’m not sure if it’s the best approach:
Maybe this issue helps explain this behavior (I also found other complaints similar to this for projects other than Argo), if so I think we would add a conditional check before casting the object:
I’m assuming with this change the defer will be executed and a new instance of the watcher will be executed.
Does it make sense to you @agilgur5 ?
I updated the code to try to register the “invalid object” that is causing the error. What is happening is that the event variable is null. I don’t have another error log. I’m using AWS EKS, so I also checked its log and I have no errors.
So I have no idea about this 😦