seaweedfs: Input/output error on fuse mount when volume becomes unwritable

Describe the bug Setup fuse mount and run some file creation test with dd command, dd will hit Input/output error at regular intervals, when writing about 20GB to 50GB data, 1000-2000 files.

System Setup

  • List the command line to start “weed master”, “weed volume”, “weed filer”, “weed s3”, “weed mount”.
 /usr/bin/weed -logtostderr=true master -mdir=/data -ip=xxx -volumePreallocate=false -volumeSizeLimitMB=4096 -defaultReplication=000
/usr/bin/weed -logtostderr=true volume -dir=/data -dir=/mnt/weed1 -mserver=localhost:9333 -max=0 -port=8080 -metricsPort=9325 -preStopSeconds=1
/usr/bin/weed -logtostderr=true volume -dir=/data -dir=/mnt/weed2 -mserver=localhost:9333 -max=0 -port=8081 -metricsPort=9326 -preStopSeconds=1
/usr/bin/weed -logdir=/var/log/udm filer -defaultReplicaPlacement=000 -master=localhost:9333 -metricsPort=9328 -maxMB=128
weed -v=4 mount -filer=localhost:8888 -dir=/opt/udm/fs1 -cacheCapacityMB=0 -concurrentWriters=8 -filer.path=/opt/udm/fs1 -collection=user1 -chunkSizeLimitMB=8
  • OS version Rocky Linux release 9.4

  • output of weed version version 30GB 3.68 linux amd64

  • if using filer, show the content of filer.toml

[mysql2]
enabled = true
createTable = """
  CREATE TABLE IF NOT EXISTS `%s` (
    `dirhash`   BIGINT NOT NULL,
    `name`      VARCHAR(766) NOT NULL,
    `directory` TEXT NOT NULL,
    `meta`      LONGBLOB,
    PRIMARY KEY (`dirhash`, `name`)
  ) DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;
"""
hostname = "127.0.0.1"
port = 3306
username = "root"
password = "xxxxx"
database = "seaweedfs"              # create or use an existing database
connection_max_idle = 2
connection_max_open = 100
connection_max_lifetime_seconds = 0
interpolateParams = false
# if insert/upsert failing, you can disable upsert or update query syntax to match your RDBMS syntax:
enableUpsert = true
upsertQuery = """INSERT INTO `%s` (`dirhash`,`name`,`directory`,`meta`) VALUES (?,?,?,?) AS `new` ON DUPLICATE KEY UPDATE `meta` = `new`.`meta`"""

Expected behavior Creating files using dd or similar tool should not hit any errors

Screenshots When hiting the error, I got below logs

dd errors:

run: dd if=/dev/mapper/rl-root of=/opt/udm/fs1/test03/20240828_1615_334/253.file.dat bs=32KB count=128 skip=2300
128+0 records in

run: dd if=/dev/mapper/rl-root of=/opt/udm/fs1/test03/20240828_1615_334/254.file.dat bs=32KB count=4096 skip=3931
dd: closing output file '/opt/udm/fs1/test03/20240828_1615_334/254.file.dat': Input/output error
run: dd if=/dev/mapper/rl-root of=/opt/udm/fs1/test03/20240828_1615_334/255.file.dat bs=32KB count=64 skip=2815
^C^C^C^Cdd: closing output file '/opt/udm/fs1/test03/20240828_1615_334/255.file.dat': Input/output error
run: dd if=/dev/mapper/rl-root of=/opt/udm/fs1/test03/20240828_1615_334/256.file.dat bs=32KB count=64 skip=1915
dd: closing output file '/opt/udm/fs1/test03/20240828_1615_334/256.file.dat': Input/output error
run: dd if=/dev/mapper/rl-root of=/opt/udm/fs1/test03/20240828_1615_334/257.file.dat bs=32KB count=8 skip=1189
dd: closing output file '/opt/udm/fs1/test03/20240828_1615_334/257.file.dat': Input/output error
run: dd if=/dev/mapper/rl-root of=/opt/udm/fs1/test03/20240828_1615_334/258.file.dat bs=32KB count=1 skip=2273
dd: closing output file '/opt/udm/fs1/test03/20240828_1615_334/258.file.dat': Input/output error
run: dd if=/dev/mapper/rl-root of=/opt/udm/fs1/test03/20240828_1615_334/259.file.dat bs=32KB count=128 skip=3628

fuse mount log:

I0828 16:15:50.182690 weedfs_file_mkrm.go:85 mknod: directory:"/opt/udm/fs1/test03/20240828_1615_334" entry:{name:"254.file.dat" attributes:{mtime:1724832950 file_mode:420 crtime:1724832950 inode:9692435665168868916}} signatures:-2101554117 skip_check_parent_directory:true
I0828 16:15:50.186464 weedfs_file_mkrm.go:98 mknod /opt/udm/fs1/test03/20240828_1615_334/254.file.dat: <nil>
I0828 16:15:50.186567 weedfs_file_sync.go:98 doFlush /opt/udm/fs1/test03/20240828_1615_334/254.file.dat fh 854

...

I0828 16:15:50.476280 dirty_pages_chunked.go:82 /opt/udm/fs1/test03/20240828_1615_334/254.file.dat saveToStorage 33,44238a82cf08 [41943040,50331648)

I0828 16:15:50.501830 dirty_pages_chunked.go:82 /opt/udm/fs1/test03/20240828_1615_334/254.file.dat saveToStorage 33,4424add8cb8a [58720256,67108864)

I0828 16:15:50.511887 dirty_pages_chunked.go:82 /opt/udm/fs1/test03/20240828_1615_334/254.file.dat saveToStorage 33,44219ea3f047 [50331648,58720256)

I0828 16:15:50.522943 dirty_pages_chunked.go:82 /opt/udm/fs1/test03/20240828_1615_334/254.file.dat saveToStorage 33,442221056f72 [33554432,41943040)
I0828 16:15:50.522949 upload_pipeline.go:191  uploaderCount 2 --> 1
I0828 16:15:50.522954 upload_pipeline.go:37 Free sealed chunk:  finished uploading chunk 4

I0828 16:15:50.537596 dirty_pages_chunked.go:82 /opt/udm/fs1/test03/20240828_1615_334/254.file.dat saveToStorage 33,44250b672f2c [92274688,100663296)
I0828 16:15:50.537617 upload_pipeline.go:191  uploaderCount 1 --> 0
I0828 16:15:50.537625 upload_pipeline.go:37 Free sealed chunk:  finished uploading chunk 11


I0828 16:15:50.584757 weedfs_file_sync.go:98 doFlush /opt/udm/fs1/test03/20240828_1615_334/254.file.dat fh 854
I0828 16:15:50.584764 upload_pipeline.go:171  uploaderCount 2 ++> 3
I0828 16:15:50.584772 upload_pipeline.go:171  uploaderCount 3 ++> 4
I0828 16:15:50.584775 upload_pipeline.go:171  uploaderCount 4 ++> 5
I0828 16:15:50.584778 upload_pipeline.go:171  uploaderCount 5 ++> 6
I0828 16:15:50.584780 upload_pipeline.go:171  uploaderCount 6 ++> 7
I0828 16:15:50.584782 upload_pipeline.go:171  uploaderCount 7 ++> 8
I0828 16:15:50.584784 upload_pipeline.go:171  uploaderCount 8 ++> 9
I0828 16:15:50.591174 dirty_pages_chunked.go:82 /opt/udm/fs1/test03/20240828_1615_334/254.file.dat saveToStorage 33,4427a57ed4d1 [25165824,33554432)
I0828 16:15:50.591205 upload_pipeline.go:191  uploaderCount 9 --> 8
I0828 16:15:50.591213 upload_pipeline.go:37 Free sealed chunk:  finished uploading chunk 3
I0828 16:15:50.591232 upload_pipeline.go:171  uploaderCount 8 ++> 9
I0828 16:15:50.634069 dirty_pages_chunked.go:82 /opt/udm/fs1/test03/20240828_1615_334/254.file.dat saveToStorage 33,4426508635e7 [67108864,75497472)
I0828 16:15:50.634094 upload_pipeline.go:191  uploaderCount 9 --> 8
I0828 16:15:50.634100 upload_pipeline.go:37 Free sealed chunk:  finished uploading chunk 8
I0828 16:15:50.634109 upload_pipeline.go:171  uploaderCount 8 ++> 9
I0828 16:15:52.334456 weedfs_stats.go:35 reading filer stats: collection:"user1" ttl:"0s"
I0828 16:15:52.335172 weedfs_stats.go:41 read filer stats: total_size:408021893120 used_size:90112109568 file_count:17314
I0828 16:16:00.622199 wfs_filer_client.go:31 WithFilerClient 0 localhost:18888: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622214 wfs_filer_client.go:31 WithFilerClient 0 localhost:18888: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622222 weedfs_write.go:47 upload data 254.file.dat: filerGrpcAddress assign volume: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622230 weedfs_write.go:47 upload data 254.file.dat: filerGrpcAddress assign volume: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622235 wfs_filer_client.go:31 WithFilerClient 0 localhost:18888: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622242 wfs_filer_client.go:31 WithFilerClient 0 localhost:18888: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622250 weedfs_write.go:47 upload data 254.file.dat: filerGrpcAddress assign volume: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622258 weedfs_write.go:47 upload data 254.file.dat: filerGrpcAddress assign volume: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622232 wfs_filer_client.go:31 WithFilerClient 0 localhost:18888: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622262 dirty_pages_chunked.go:76 /opt/udm/fs1/test03/20240828_1615_334/254.file.dat saveToStorage [125829120,131072000): upload data: filerGrpcAddress assign volume: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622264 weedfs_write.go:47 upload data 254.file.dat: filerGrpcAddress assign volume: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622266 dirty_pages_chunked.go:76 /opt/udm/fs1/test03/20240828_1615_334/254.file.dat saveToStorage [83886080,92274688): upload data: filerGrpcAddress assign volume: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622267 upload_pipeline.go:191  uploaderCount 9 --> 8
I0828 16:16:00.622269 upload_pipeline.go:191  uploaderCount 8 --> 7
I0828 16:16:00.622258 dirty_pages_chunked.go:76 /opt/udm/fs1/test03/20240828_1615_334/254.file.dat saveToStorage [16777216,25165824): upload data: filerGrpcAddress assign volume: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622272 upload_pipeline.go:37 Free sealed chunk:  finished uploading chunk 15
I0828 16:16:00.622274 upload_pipeline.go:191  uploaderCount 7 --> 6
I0828 16:16:00.622277 upload_pipeline.go:37 Free sealed chunk:  finished uploading chunk 2
I0828 16:16:00.622213 wfs_filer_client.go:31 WithFilerClient 0 localhost:18888: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622283 weedfs_write.go:47 upload data 254.file.dat: filerGrpcAddress assign volume: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622286 dirty_pages_chunked.go:76 /opt/udm/fs1/test03/20240828_1615_334/254.file.dat saveToStorage [8388608,16777216): upload data: filerGrpcAddress assign volume: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622288 upload_pipeline.go:191  uploaderCount 6 --> 5
I0828 16:16:00.622291 upload_pipeline.go:37 Free sealed chunk:  finished uploading chunk 1
I0828 16:16:00.622238 dirty_pages_chunked.go:76 /opt/udm/fs1/test03/20240828_1615_334/254.file.dat saveToStorage [75497472,83886080): upload data: filerGrpcAddress assign volume: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!
I0828 16:16:00.622294 upload_pipeline.go:191  uploaderCount 5 --> 4
I0828 16:16:00.622295 upload_pipeline.go:37 Free sealed chunk:  finished uploading chunk 9
I0828 16:16:00.622230 dirty_pages_chunked.go:76 /opt/udm/fs1/test03/20240828_1615_334/254.file.dat saveToStorage [117440512,125829120): upload data: filerGrpcAddress assign volume: assign volume failure count:1 collection:"user1" path:"/opt/udm/fs1/test03/20240828_1615_334/254.file.dat": assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:user1 replication:000 ttl: error: No more writable volumes!

master log at same time:

I0828 08:15:50.581606 volume_layout.go:405 Volume 33 becomes unwritable
I0828 08:15:50.581623 volume_layout.go:497 Volume 33 reaches full capacity.

I0828 08:16:55.527601 volume_layout.go:418 Volume 36 becomes writable
I0828 08:16:55.527606 volume_growth.go:270 Registered Volume 36 on topo:DefaultDataCenter:DefaultRack:192.168.3.138:8081
I0828 08:16:55.630887 volume_growth.go:257 Created Volume 37 on topo:DefaultDataCenter:DefaultRack:192.168.3.138:8080
I0828 08:16:55.630903 volume_layout.go:418 Volume 37 becomes writable
I0828 08:16:55.630927 volume_growth.go:270 Registered Volume 37 on topo:DefaultDataCenter:DefaultRack:192.168.3.138:8080


volume id 33 status:

    {
      "Id": 33,
      "Size": 5233556081,
      "ReplicaPlacement": {},
      "Ttl": {
        "Count": 0,
        "Unit": 0
      },
      "DiskType": "",
      "Collection": "user1",
      "Version": 3,
      "FileCount": 1031,
      "DeleteCount": 0,
      "DeletedByteCount": 0,
      "ReadOnly": true,
      "CompactRevision": 0,
      "ModifiedAtSecond": 0,
      "RemoteStorageName": "",
      "RemoteStorageKey": ""
    },

filer status for last succeeded file and first failed file:

    {
      "FullPath": "/opt/udm/fs1/test03/20240828_1615_334/253.file.dat",
      "Mtime": "2024-08-28T08:15:50Z",
      "Crtime": "2024-08-28T08:15:50Z",
      "Mode": 420,
      "Uid": 0,
      "Gid": 0,
      "Mime": "",
      "TtlSec": 0,
      "UserName": "",
      "GroupNames": null,
      "SymlinkTarget": "",
      "Md5": null,
      "FileSize": 4096000,
      "Rdev": 0,
      "Inode": 12258707599359591449,
      "Extended": null,
      "chunks": [
        {
          "file_id": "33,44203cf3c907",
          "size": 4096000,
          "modified_ts_ns": 1724832950141367616,
          "e_tag": "RsxScYUTchacTIVF4Yz/Mg==",
          "fid": {
            "volume_id": 33,
            "file_key": 17440,
            "cookie": 1022609671
          },
          "is_compressed": true
        }
      ],
      "HardLinkId": null,
      "HardLinkCounter": 0,
      "Content": null,
      "Remote": null,
      "Quota": 0
    },
    {
      "FullPath": "/opt/udm/fs1/test03/20240828_1615_334/254.file.dat",
      "Mtime": "2024-08-28T08:15:50Z",
      "Crtime": "2024-08-28T08:15:50Z",
      "Mode": 420,
      "Uid": 0,
      "Gid": 0,
      "Mime": "",
      "TtlSec": 0,
      "UserName": "",
      "GroupNames": null,
      "SymlinkTarget": "",
      "Md5": null,
      "FileSize": 0,
      "Rdev": 0,
      "Inode": 9692435665168868916,
      "Extended": null,
      "HardLinkId": null,
      "HardLinkCounter": 0,
      "Content": null,
      "Remote": null,
      "Quota": 0
    },


Additional context It looks like filer has problem choosing volume for write, volume 33 here is apparently full after writing 253.file.dat, but it seems when writing 254.file.dat, filer side has no knowledge of that, and still using volume 33 for uploading, until volume 33 becomes unwritable (readonly), and the write has to fail.

I think basically there are 2 approaches:

  • improve how filer choose the right volume for write
  • retry the failed write to new volume

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 31 (21 by maintainers)

Most upvoted comments

I’ve tested the pr 5884 in our environment, it can grow volumes fine.