moby: Error trying to remove dead nodes from swarm: raft message is too large and can't be sent
I have a swarm with ~1300 nodes, and some enter and leave all the time (about 10/minute).
Since about a week ago, I’m experiencing an error when trying to remove dead nodes with docker node rm xxxx
from the swarm:
Error response from daemon: rpc error: code = Unknown desc = raft: raft message is too large and can't be sent
All I see in the logs is the same:
Apr 5 12:35:31 ip-10-0-0-10 dockerd[1239]: time="2018-04-05T12:35:31.686809606Z" level=error msg="Error removing node x2nhsvmnzoaq5hp3xqfl2a7dp: rpc error: code = Unknown desc = raft: raft message is too large and can't be sent"
Apr 5 12:35:31 ip-10-0-0-10 dockerd[1239]: time="2018-04-05T12:35:31.687236652Z" level=error msg="Handler for DELETE /v1.37/nodes/x2nhsvmnzoaq5hp3xqfl2a7dp returned error: rpc error: code = Unknown desc = raft: raft message is too large and can't be sent"
Apr 5 12:35:36 ip-10-0-0-10 dockerd[1239]: time="2018-04-05T12:35:36.289574281Z" level=error msg="Error removing node kdmprxylwjmvutsfb9y1f2o17: rpc error: code = Unknown desc = raft: raft message is too large and can't be sent"
Apr 5 12:35:36 ip-10-0-0-10 dockerd[1239]: time="2018-04-05T12:35:36.289644704Z" level=error msg="Handler for DELETE /v1.37/nodes/kdmprxylwjmvutsfb9y1f2o17 returned error: rpc error: code = Unknown desc = raft: raft message is too large and can't be sent"
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 16 (8 by maintainers)
@eduardolundgren the root cause of your issue is very different, but I know what the problem is. I’m opening an issue on swarmkit, docker/swarmkit#2655