prometheus-operator: grafana-dashboards ConfigMap is invalid (metadata.annotations too long)
What did you do? Added dashboards to assets directory and recreate + apply the ConfigMap
What did you expect to see? ConfigMap updated and applied.
What did you see instead? Under which circumstances? Error in ConfigMap: The ConfigMap “grafana-dashboards” is invalid: metadata.annotations: Too long: must have at most 262144 characters
Environment
-
Kubernetes version information: 1.7 and 1.6 (tested with both)
-
Kubernetes cluster kind:
kops towards AWS
- Manifests: Not much relevant, it’s a matter of size. If I add only one or 2 .json files it works fine, but when I try to add the 5 json files I have it fails. Here I paste the output…
$ ls -1 assets/grafana/
all-nodes-dashboard.json
deployment-dashboard.json
kafka-brokers-dashboard.json
kafka-brokers-topics-dashboard.json
kafka-jvm-dashboard.json
kafka-streams-dashboard.json
kafka-zookeepers-dashboard.json
kubernetes-pods-dashboard.json
node-dashboard.json
prometheus-datasource.json
resource-requests-dashboard.json
$ make generate
>> Compiling assets and generating Kubernetes manifests
$ kubectl -n monitoring apply -f manifests/grafana/grafana-dashboards.yaml
The ConfigMap "grafana-dashboards" is invalid: metadata.annotations: Too long: must have at most 262144 characters
346803 bytes --> manifests/grafana/grafana-dashboards.yaml
This looks like a clear kubernetes limit… but I don’t fully understand it, and the generated ConfigMap manifest is not that big… What would you suggest?
Is there a way to increase that limit?
Thanks!
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 42 (35 by maintainers)
Sorry to come back here, but I still have the exact same issue as mentioned here with latest
prometheus-operatorusing theHelmchart deployment. I’m on GKE-1.10.5/3I’m wondering why we’re not using a
PVCto hold the dashboards instead of aConfigfile?Thanks to your hints, I have realized that the size limit reached in this case is the “annotations” size limit of kubernetes, which is 256 KB and not the ConfigMap limit which is 1MB.
That limit was discussed here: https://github.com/kubernetes/kubernetes/issues/15878
The only way I have found to “skip” the annotation is issuing a
kubectl replaceinstead ofkubectl apply. That works perfect (because my manifest is smaller than 1MB), but as you said we lose the “rollback” option (I would assume that the user of this should have the dashboards under control and if something is lost, he could load a previous version of the yaml somehow).I will test later the “bin pack” proposal. If I have understood correctly, you mean creating ConfigMaps like grafana-dashboards-0, grafana-dashboards-1, … each of them with less than 1MB, right? Are you confident that will work? 😃 or should they be called “grafana-dashboards0”, “grafana-dashboards1”, … ?
With this bin-packing… can I really “break” a json file into 2 different ConfigMaps? If so then I could create a tool with a configurable limit to write multiple ConfigMaps based on the json files + the configured limit. Or if I always have to start and end the ConfigMap with complete json files let me know to make it accordingly.
Yes, @brancz , I will prepare it as part of kube-prometheus, let me work on that and i will propose a PR soon for your validation.
I think the bug is still present in @eedugon generator. By using the script contrib/kube-prometheus/hack/scripts/generate-manifests.sh it generate a configmap that I wasn’t able to deploy (same error, metadata.annotations too long) By tuning the “-s” value to a lower number (150000) I was able to generate a proper configmap file.
got it working… I will update the PR
If it’s a quick one I’d say let’s complete it now, while we’re already at it.
Absolutely no worries! I just got back from vacation myself and slowly working through things, take the time off of GitHub 🙂 burnout is a real thing.
Ok, I understand. Then for the moment, and as a workaround, we could use kubectl
replaceinstead ofapplyfor the dashboards in case of problems (as if we use replace the limit will become 1MB instead of 256KB).We can keep this tool on hold until it’s really necessary. For the moment I will configure a big limit size in the tool and then it will always generate single configmaps. If you want me to update or share the tool somehow for your review let me know. Currently is a standalone small tool, but the algorithm could be integrated easily inside one of the hack/scripts/ files to be handled by
make generate.But if you want to move in another direction let me know. Currently my coding skills are mainly bash, so I don’t think I can help on updating or improving grafana-watcher, unless is written in bash =)
mmm sure, thanks for the hints. I will take a deeper look and write back the conclusions.