seaweedfs: WeedFS can't start normally caused by Raft Server fault.

Describe the bug Some time server cannot start normally because of raft server may down?

System Setup I started this server with docker-compose

version: '3'

# weed/redis/...'s exposed port is only for debug

services:
  master:
    image: chrislusf/seaweedfs
    ports:
      - 9333:9333 # todo remove this port expose
    command: "master -mdir /data"
    volumes:
      - "./data/meta/:/data/"
  volume:
    image: chrislusf/seaweedfs
    ports:
      - 9301:9301 # todo remove this port expose
    command: 'volume -max=30 -mserver="master:9333" -port=9301'
    volumes:
      - "./data/volume/:/data/"
    depends_on:
      - master
  filer:
    image: chrislusf/seaweedfs
    ports:
      - 9401:9401 # todo remove this port expose
    command: 'filer -master="master:9333" -port=9401 -dirListLimit 1000000'
    volumes:
      - "./data/filer/:/data/filerldb2/"
    depends_on:
      - master
      - volume
  redis:
    image: redis:5.0
    ports:
      - 6379:6379 # todo remove this port expose

I’m using default filer.toml, but I think this have no relationship with filer.

Expected behavior Start normally

Screenshots

These are logs repeately printed:

.....
volume_1  | I1114 08:21:11     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
master_1  | I1114 08:21:11     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:72] added volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:102] master see new volume 1 from 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:102] master see new volume 2 from 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:102] master see new volume 6 from 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:102] master see new volume 3 from 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:102] master see new volume 5 from 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:102] master see new volume 7 from 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:102] master see new volume 4 from 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:24] unregister disconnected volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 topology_event_handling.go:57] Removing Volume 3 from the dead volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 volume_layout.go:261] Volume 3 has 0 replica, less than required 1
master_1  | I1114 08:21:11     1 volume_layout.go:237] Volume 3 becomes unwritable
master_1  | I1114 08:21:11     1 topology_event_handling.go:57] Removing Volume 5 from the dead volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 volume_layout.go:261] Volume 5 has 0 replica, less than required 1
master_1  | I1114 08:21:11     1 volume_layout.go:237] Volume 5 becomes unwritable
master_1  | I1114 08:21:11     1 topology_event_handling.go:57] Removing Volume 7 from the dead volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 volume_layout.go:261] Volume 7 has 0 replica, less than required 1
master_1  | I1114 08:21:11     1 volume_layout.go:237] Volume 7 becomes unwritable
master_1  | I1114 08:21:11     1 topology_event_handling.go:57] Removing Volume 4 from the dead volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 volume_layout.go:261] Volume 4 has 0 replica, less than required 1
master_1  | I1114 08:21:11     1 volume_layout.go:237] Volume 4 becomes unwritable
master_1  | I1114 08:21:11     1 topology_event_handling.go:57] Removing Volume 1 from the dead volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 volume_layout.go:261] Volume 1 has 0 replica, less than required 1
master_1  | I1114 08:21:11     1 volume_layout.go:237] Volume 1 becomes unwritable
master_1  | I1114 08:21:11     1 topology_event_handling.go:57] Removing Volume 2 from the dead volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 volume_layout.go:261] Volume 2 has 0 replica, less than required 1
master_1  | I1114 08:21:11     1 volume_layout.go:237] Volume 2 becomes unwritable
master_1  | I1114 08:21:11     1 topology_event_handling.go:57] Removing Volume 6 from the dead volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 volume_layout.go:261] Volume 6 has 0 replica, less than required 1
volume_1  | I1114 08:21:11     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
master_1  | I1114 08:21:11     1 volume_layout.go:237] Volume 6 becomes unwritable
master_1  | I1114 08:21:11     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.22.0.4:9301
master_1  | E1114 08:21:12     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1114 08:21:12     1 masterclient.go:88] master failed to receive from 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1114 08:21:12     1 masterclient.go:116] master failed to connect with master 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1114 08:21:12     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
filer_1   | I1114 08:21:12     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1114 08:21:12     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1114 08:21:13     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1114 08:21:13     1 masterclient.go:88] master failed to receive from 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1114 08:21:13     1 masterclient.go:116] master failed to connect with master 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1114 08:21:13     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
filer_1   | I1114 08:21:13     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1114 08:21:13     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1114 08:21:14     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1114 08:21:14     1 masterclient.go:88] master failed to receive from 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1114 08:21:14     1 masterclient.go:116] master failed to connect with master 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1114 08:21:14     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
filer_1   | I1114 08:21:14     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1114 08:21:14     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1114 08:21:15     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1114 08:21:15     1 masterclient.go:88] master failed to receive from 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1114 08:21:15     1 masterclient.go:116] master failed to connect with master 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1114 08:21:15     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1114 08:21:15     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1114 08:21:15     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
......

Additional context None for now

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15

Commits related to this issue

set master ip in docker compose yaml file fix https://github.com/chrislusf/seaweedfs/issues/1118 — committed to seaweedfs/seaweedfs by chrislusf 5 years ago
treat it as a single node cluster if empty raft server name possible fix for https://github.com/chrislusf/seaweedfs/issues/1118 — committed to seaweedfs/seaweedfs by chrislusf 4 years ago

Most upvoted comments

I use same docker-compose file but add grpc port ,when restart the cluster,same error

master_1  | I1115 16:48:02     1 master_grpc_server.go:72] added volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 topology_event_handling.go:57] Removing Volume 3 from the dead volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 volume_layout.go:261] Volume 3 has 0 replica, less than required 1
master_1  | I1115 16:48:02     1 volume_layout.go:237] Volume 3 becomes unwritable
master_1  | I1115 16:48:02     1 topology_event_handling.go:57] Removing Volume 1 from the dead volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 volume_layout.go:261] Volume 1 has 0 replica, less than required 1
master_1  | I1115 16:48:02     1 volume_layout.go:237] Volume 1 becomes unwritable
master_1  | I1115 16:48:02     1 topology_event_handling.go:57] Removing Volume 2 from the dead volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 volume_layout.go:261] Volume 2 has 0 replica, less than required 1
master_1  | I1115 16:48:02     1 volume_layout.go:237] Volume 2 becomes unwritable
master_1  | I1115 16:48:02     1 topology_event_handling.go:57] Removing Volume 4 from the dead volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 volume_layout.go:261] Volume 4 has 0 replica, less than required 1
master_1  | I1115 16:48:02     1 volume_layout.go:237] Volume 4 becomes unwritable
master_1  | I1115 16:48:02     1 topology_event_handling.go:57] Removing Volume 5 from the dead volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 volume_layout.go:261] Volume 5 has 0 replica, less than required 1
master_1  | I1115 16:48:02     1 volume_layout.go:237] Volume 5 becomes unwritable
master_1  | I1115 16:48:02     1 topology_event_handling.go:57] Removing Volume 6 from the dead volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 volume_layout.go:261] Volume 6 has 0 replica, less than required 1
master_1  | I1115 16:48:02     1 volume_layout.go:237] Volume 6 becomes unwritable
master_1  | I1115 16:48:02     1 topology_event_handling.go:57] Removing Volume 7 from the dead volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 volume_layout.go:261] Volume 7 has 0 replica, less than required 1
master_1  | I1115 16:48:02     1 volume_layout.go:237] Volume 7 becomes unwritable

xiaoweihong on Nov 15, 2019