moby: Error trying to remove dead nodes from swarm: raft message is too large and can't be sent

I have a swarm with ~1300 nodes, and some enter and leave all the time (about 10/minute).

Since about a week ago, I’m experiencing an error when trying to remove dead nodes with docker node rm xxxx from the swarm:

Error response from daemon: rpc error: code = Unknown desc = raft: raft message is too large and can't be sent

All I see in the logs is the same:

Apr  5 12:35:31 ip-10-0-0-10 dockerd[1239]: time="2018-04-05T12:35:31.686809606Z" level=error msg="Error removing node x2nhsvmnzoaq5hp3xqfl2a7dp: rpc error: code = Unknown desc = raft: raft message is too large and can't be sent"
Apr  5 12:35:31 ip-10-0-0-10 dockerd[1239]: time="2018-04-05T12:35:31.687236652Z" level=error msg="Handler for DELETE /v1.37/nodes/x2nhsvmnzoaq5hp3xqfl2a7dp returned error: rpc error: code = Unknown desc = raft: raft message is too large and can't be sent"
Apr  5 12:35:36 ip-10-0-0-10 dockerd[1239]: time="2018-04-05T12:35:36.289574281Z" level=error msg="Error removing node kdmprxylwjmvutsfb9y1f2o17: rpc error: code = Unknown desc = raft: raft message is too large and can't be sent"
Apr  5 12:35:36 ip-10-0-0-10 dockerd[1239]: time="2018-04-05T12:35:36.289644704Z" level=error msg="Handler for DELETE /v1.37/nodes/kdmprxylwjmvutsfb9y1f2o17 returned error: rpc error: code = Unknown desc = raft: raft message is too large and can't be sent"

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 16 (8 by maintainers)

Most upvoted comments

@eduardolundgren the root cause of your issue is very different, but I know what the problem is. I’m opening an issue on swarmkit, docker/swarmkit#2655