seaweedfs: [BUG] - Weed masters show different results when look up a volumeId - error volume id not found

Hi @chrislusf , After facing a lot of errors as follows, I lookup “volumeId” manually on all masters, and this is the result I got: “error volume id {volumeId} not found”

On Master one(That is the leader): 1.

curl http://weed-master-1.local:9333/dir/lookup?volumeId=1662

result:

{"volumeOrFileId":"1662","error":"volume id 1662 not found"}

On master two: 2.

http://weed-master-2.local:9333/dir/lookup?volumeId=1662

result:

{

    "volumeOrFileId": "1662",
    "locations": [
        {
            "url": "weed-volume-003.local:8082",
            "publicUrl": "weed-volume-003.local:8082"
        },
        {
            "url": "weed-volume-005.local:8080",
            "publicUrl": "weed-volume-005.local:8080"
        }
    ]

}

On master three:

curl http://weed-master-3.local:9333/dir/lookup?volumeId=1662

{

    "volumeOrFileId": "1662",
    "locations": [
        {
            "url": "weed-volume-003.local:8082",
            "publicUrl": "weed-volume-003.local:8082"
        },
        {
            "url": "weed-volume-005.local:8080",
            "publicUrl": "weed-volume-005.local:8080"
        }
    ]

}

I found out this problem will cause due to an unknown reason when the leader changes and after that, this problem would happen, and it will be solved if I restart all three masters at the same time . This log is for the moment that master changed and errors began to start:

Mar 13 13:53:53 weed-master-1 seaweedfs-master[1121]: I0313 13:53:53  1121 master_server.go:165] leader change event: weed-master-2.local:9333 =>
Mar 13 13:53:53 weed-master-1 seaweedfs-master[1121]: I0313 13:53:53  1121 master_server.go:165] leader change event:  => weed-master-3.local:9333
Mar 13 13:53:53 weed-master-1 seaweedfs-master[1121]: I0313 13:53:53  1121 master_server.go:168] [ weed-master-1.local:9333 ] weed-master-3.local:9333 becomes leader.
Mar 13 13:54:15 weed-master-1 seaweedfs-master[1121]: I0313 13:54:15  1121 masterclient.go:128] master masterClient failed to receive from weed-master-2.local:9333: rpc error: code = Unavailable desc = error reading from server: EOF
Mar 13 13:54:15 weed-master-1 seaweedfs-master[1121]: I0313 13:54:15  1121 common.go:70] response method:GET URL:/dir/lookup?volumeId=2243 with httpStatus:404 and JSON:{"volumeOrFileId":"2243","error":"volume id 2243 not found"}
Mar 13 13:54:15 weed-master-1 seaweedfs-master[1121]: I0313 13:54:15  1121 common.go:70] response method:GET URL:/dir/lookup?volumeId=5817 with httpStatus:404 and JSON:{"volumeOrFileId":"5817","error":"volume id 5817 not found"}
Mar 13 13:54:15 weed-master-1 seaweedfs-master[1121]: I0313 13:54:15  1121 common.go:70] response method:GET URL:/dir/lookup?volumeId=243 with httpStatus:404 and JSON:{"volumeOrFileId":"243","error":"volume id 243 not found"}
Mar 13 13:54:15 weed-master-1 seaweedfs-master[1121]: I0313 13:54:15  1121 common.go:70] response method:GET URL:/dir/lookup?volumeId=1827 with httpStatus:404 and JSON:{"volumeOrFileId":"1827","error":"volume id 1827 not found"}
Mar 13 13:54:15 weed-master-1 seaweedfs-master[1121]: I0313 13:54:15  1121 common.go:70] response method:GET URL:/dir/lookup?volumeId=1902 with httpStatus:404 and JSON:{"volumeOrFileId":"1902","error":"volume id 1902 not found"}
Mar 13 13:54:15 weed-master-1 seaweedfs-master[1121]: I0313 13:54:15  1121 common.go:70] response method:GET URL:/dir/lookup?volumeId=5434 with httpStatus:404 and JSON:{"volumeOrFileId":"5434","error":"volume id 5434 not found"}
Mar 13 13:54:15 weed-master-1 seaweedfs-master[1121]: I0313 13:54:15  1121 common.go:70] response method:GET URL:/dir/lookup?volumeId=1636 with httpStatus:404 and JSON:{"volumeOrFileId":"1636","error":"volume id 1636 not found"}

seaweedfs version: version 8000GB 2.93 f7ee60996c433a596b0ea75961cc5f23c6663c6f linux amd64 (I have this problem from weed version 2.89, I upgrade to 2.93 but the bug still exists )