kubernetes: Bug: ResourceExhausted desc = grpc: received message larger than max (4195017 vs. 4194304)

For anyone operating a k8s cluster, running into this error and not being sure what the actual problem is… If you run this docker inspect $(docker ps -a | cut -d ' ' -f1 | tail -n+2) | wc -c, you’ll see that a message describing all containers currently known to docker is bigger than the limit the Kubelet has set for gRPC messages (16MB at the time of writing).

This either means you have a LOT of (possibly dead) containers, or there’s a LOT of metadata attached to your containers. This metadata probably comes in the form of custom labels attached to the container image by whatever tool you’re using to create the container images. Check out docker describe for a couple of your containers and the issue should become apparent.

@dims yes it would work but it’s 1GB so we are in a runaway situation here; have 1GB today and you’ll need 2GB tomorrow. 16MB is definitely an improvement and your change to 16MB is LGTM (thanks!), but the root problem still remains. The 2 solutions I can imagine are (1) an API change to include paging (or streaming), or (2) make the kubelet ensure there’s less than a total number of containers (running+exited).

Since the issue also impacts ctr I’m participating in the following upstream issue: https://github.com/containerd/containerd/issues/2320.

@dims I can already tell that it probably won’t solve the problem (with containerd at least):

# ctr -n k8s.io c ls
ctr: grpc: trying to send message larger than max (925727095 vs. 16777216): unknown

The solution here for me, install nerdctl, then:

sudo nerdctl -n k8s.io container prune
sudo systemctl restart containerd

Once containers are pruned, you need to restart containerd for the node to show ready again.

@oldthreefeng I’m glad it helped.

In my case, after adding --maximum-dead-containers=1000, it still happened once after.

So I have to add a docker-prune service for double insurance.

Just for reference

/etc/systemd/system/docker-prune.service

[Unit]
Description=Docker System prune everything

[Service]
Type=oneshot
ExecStart=/usr/bin/sh -c '/usr/bin/docker system prune -a --force'

[Install]
WantedBy=multi-user.target

/etc/systemd/system/docker-prune.timer

[Unit]
Description=Run docker-prune Weekly on Sunday

[Timer]
OnCalendar=Sun

[Install]
WantedBy=multi-user.target

It will run once a week on Sunday Using systemctl enable and start both. check the status with systemctl list-timers.

My case was caused by too many tekton dead pod (over 3000 dead containers). I will try to add --maximum-dead-containers=1000 to the kubelet to solve the issue https://kubernetes.io/docs/concepts/architecture/garbage-collection/#container-image-garbage-collection https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

Thanks @MG2R your comment was really helpful!! However, is there any systematic solution to resolve this? rather than docker system prune on such node or remove dead containers manually? I mean, is there any chance to increase limit of gRPC message size?

@dims yes it would work but it’s 1GB so we are in a runaway situation here; have 1GB today and you’ll need 2GB tomorrow. 16MB is definitely an improvement and your change to 16MB is LGTM (thanks!), but the root problem still remains. The 2 solutions I can imagine are (1) an API change to include paging (or streaming), or (2) make the kubelet ensure there’s less than a total number of containers (running+exited).

Since the issue also impacts ctr I’m participating in the following upstream issue: containerd/containerd#2320.

Really? There’s an absolute universe more information in 1GB that 4MB, just as it made sense to not make the limit 4096. Although the above is true, can we agree there is huge difference, in human terms, between 4MB and say 1GB.

Pretty please change the limit to a round 1GB or something that is configurable at run-time by the service publisher or as a connection option. Maybe an option flag that waves this limit at runtime?

This is a scary issue for us b/c now we have to write a segmenter/desegmenter in our client? Fear. Please let consumers of pb tech not have to do this. Everyone will just have to write their own segmenter/desegmenter hurting the pb’s rep as the one stop shop for interoperabilty.