seaweedfs: ec.balance not working

Describe the bug After recompiling the weed binary with the actual snapshot the binary fixes the last issues (problem with redirect, problem with some ec encoded files) but now i could see some of the ec.shards are not distributed in the right way (5 volume servers). for example volume 59:

bm6: 59		3146776576 Bytes	[0 1 2 3]	06 Nov 19 11:08 +0100
bm8: 59		3146776576 Bytes	[0 1 2 3 4 5 6 7 8 9 10]	06 Nov 19 11:06 +0100
bm9: 59		3146776576 Bytes	[4]	06 Nov 19 11:08 +0100
bm2: 59		3146776576 Bytes	[11 12 13]	06 Nov 19 10:29 +0100
bm7: 59		3146776576 Bytes	[5 6 7]	06 Nov 19 11:08 +0100

i dont know why those shards had been created this way (when doing ec.encode all volume servers had been up and those should be distributed equally) but i wanted to fix it with weed shell - ec.balance -force

but ec.balance -force only says:

balanceEcVolumes collections 1
balanceEcVolumes collection 
balanceEcVolumes 
bm7:8280 has 1 overlimit, moving ec shard 15.4
bm7:8280 has 1 overlimit, moving ec shard 60.6
bm2:8280 has 2 overlimit, moving ec shard 56.0
bm2:8280 has 1 overlimit, moving ec shard 56.1
bm6:8280 has 1 overlimit, moving ec shard 59.0

and seems to be doing nothing.

weed master is also configured to do:

[master.maintenance]
# periodically run these scripts are the same as running them from 'weed shell'
scripts = """
  ec.encode -fullPercent=95 -quietFor=1h
  ec.rebuild -force
  ec.balance -force
  volume.balance -force
"""
sleep_minutes = 17          # sleep minutes between each script execution

is balancing maybe running currently in the background ? how to check this ? i did not see anything in the logfiles …

System Setup weed version: version 30GB 1.44 linux amd64 but as stated recompiled with the latest git snapshot OS Version - mixed Ubuntu Server 14.04 and 18.04

Expected behavior

doing a rebalance
some sort of status message if something is running in the background - either in weed shell or in the web status screen - or in both 😉

additionally ec.rebuild says: error: disk space is not enough but on all (weed) disks there is plenty of storage free - or is here another path used - for example /tmp ?

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 25

Commits related to this issue

fix rack_not_found case fix rack_not_found in https://github.com/chrislusf/seaweedfs/issues/1111 — committed to seaweedfs/seaweedfs by chrislusf 5 years ago
avoid double counting ec slot addressing issue found in https://github.com/chrislusf/seaweedfs/issues/1111 The ec slots are already counted on the server side. — committed to seaweedfs/seaweedfs by chrislusf 5 years ago
add sortEcNodesByFreeslotsDecending and sortEcNodesByFreeslotsAscending addressing https://github.com/chrislusf/seaweedfs/issues/1111 — committed to seaweedfs/seaweedfs by chrislusf 5 years ago
passing in averageShardsPerEcNode instead of expectedTotalEcShards addressing https://github.com/chrislusf/seaweedfs/issues/1111 — committed to seaweedfs/seaweedfs by chrislusf 5 years ago

Most upvoted comments

i am trying my best 😉 never coded with go before …

i added some printf:

bm7:8280 has 1 overlimit, moving ec shard 60.6
possible nodes bm9:8280 bm6:8280 bm7:8280 bm2:8280 
test bm7:8280 bm9:8280 13
cont 3 - 3 >= 1
test bm7:8280 bm6:8280 13
cont 3 - 3 >= 1
test bm7:8280 bm7:8280 12
cont 1 - same node
test bm7:8280 bm2:8280 11
cont 3 - 3 >= 1

it seems, there is missing a node - in this case bm8 because all conditions are not met it skips all nodes and do not move at all. but what i dont understand - it should not move this shard, but delete one of the duplicates … bm2 [3,4,5] bm6 [0,1,2] bm7 [6 7 8 12 13] bm8 [6 7 8] bm9 [9 19 11] so it should remove the shards bm7.6, bm7.7 and bm7.8 to get a equal distribution.

steven2612 on Nov 12, 2019