gcsfuse: GCSFUSE hangs when verifying the content of a bucket.
We have had a number of occurrences of gcsfuse hanging in our production environment and have now managed to replicate the behavior in development. We retrieve a list filepaths/file names from a database or text file traverse the list and issue an ls -i, file and md5sum against each.
Here is the process to make the gcsfuse hang:
Running as root :-
1. Mount gcsfuse:
umount /usr/dot/dc/device/dev_aap/dev97 umount /usr/dot/dc/device/dev_aap/dev97 umount: /usr/dot/dc/device/dev_aap/dev97: not mounted mount ph-dev-images-bkt /usr/dot/dc/device/dev_aap/dev97 -t gcsfuse -o rw,_netdev,allow_other,uid=1111,gid=1111,key_file=/etc/google-svc-acnt-key.dev,log_file=/dot/dc/logs/aap/gcsfuse.log,log_format=text,debug_fuse_errors,debug_fuse,debug_fs,debug_gcs,debug_invariants,debug_mutex Calling gcsfuse with arguments: --gid 1111 --log-file /dot/dc/logs/aap/gcsfuse.log --debug_fuse_errors --debug_fs --key-file /etc/google-svc-acnt-key.dev --debug_gcs -o allow_other --log-format text --debug_fuse --debug_invariants -o rw --uid 1111 --debug_mutex ph-dev-images-bkt /usr/dot/dc/device/dev_aap/dev97 2023/01/18 10:31:12.338265 Start gcsfuse/0.41.10 (Go version go1.18.4) for app “” using mount point: /usr/dot/dc/device/dev_aap/dev97
2. Make test data ./GCS-TESTER.sh /usr/dot/dc/device/dev_aap/dev97 make-test-files … wait an hour or two
3. Check test data ./GCS-TESTER.sh /usr/dot/dc/device/dev_aap/dev97 check-test-files …
After several minutes the checking may hang after printing something like the following : Wed Jan 18 10:35:12 AEDT 2023 : count file: 1247 … checking /usr/dot/dc/device/dev_aap/dev97/device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg … -rw-r–r-- 1 digicol dc 3868762 Jan 17 13:56 /usr/dot/dc/device/dev_aap/dev97/device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg /usr/dot/dc/device/dev_aap/dev97/device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg: data
After the above happens, gcsfuse will recovers after about 1 hour. From past experience, the md5sum against the file involved in the current hang will expire after 30 mins. Then the ls -al for the next file to be tested will timeout after another 30 mins. Then gcsfuse will start working correctly, and the test script will start testing new files …
See attached https://drive.google.com/file/d/19jbPH3N1yzdRRAh9RdKz5qzyeV5eaIP6/view?usp=sharing starting from when gcsfuse was mounted above until about 1 hour later when gcsfuse seemed to recover and the tester script ceased to hang on one file and was again able to quickly test files one after another.
#########################################################
# cat GCS-TESTER.sh
#!/bin/bash
#this script tests gcsfuse running against gcs by generating test content and then checking that test content stored in a gcs mount ...
usage () {
echo "Usage : $0 <mount-point> make-test-files"
echo " : $0 <mount-point> check-test-files"
echo " : $0 <mount-point> make-and-check-test-files"
echo " eg : $0 /mnt/dev97 make-test-files"
echo " eg : $0 /mnt/dev97 check-test-files"
echo " eg : $0 /mnt/dev97 make-and-check-test-files"
echo ""
exit
}
make-dummy-test-files () {
echo "$(date) : make dummy test files ..."
for yyyy in $TESTFOLDER; do
for mm_dd in 01-06 01-07 01-08 01-09 01-10; do
for s1 in $(seq 0 255 | sort -R | head -5); do
for s2 in $(seq 0 255 | sort -R | head -5); do
for s3 in $(seq 0 255 | sort -R | head -5); do
randomstring=$(echo $(LC_ALL=C tr -dc a-z0-9 </dev/urandom | head -c $(shuf -i 19-21 -n 1)))
for type in layout thumbnail minihighres watermark original; do
#echo -n "yyyy: $yyyy mm_dd: $mm_dd s1: $s1 s2: $s2 s3: $s3 ... "
path=$(printf "%s/device/imagearc/%s/%s/%02x/%02x/%02x" $MOUNTPOINT $yyyy $mm_dd $s1 $s2 $s3)
file=${fixedstring}-${randomstring}_$type.jpg
[ $type == "thumbnail" ] && size=$(shuf -i 1500-7000 -n 1)
[ $type == "layout" ] && size=$(shuf -i 7000-46000 -n 1)
[ $type == "watermark" ] && size=$(shuf -i 20000-147000 -n 1)
[ $type == "minihighres" ] && size=$(shuf -i 15000-147000 -n 1)
[ $type == "original" ] && size=$(shuf -i 499000-17000000 -n 1)
echo "size:$size path: $path file: $file ..."
mkdir -p $path
head -c $size /dev/urandom > $path/$file
done
done
done
done
done
done
echo "$(date) : dummy test files created."
echo "$(date) : make list of files to test against ..."
find $MOUNTPOINT/device/imagearc/$TESTFOLDER/ -type f > $HOMEDIR/TEST-GCS-FILES-$TESTFOLDER.txt
echo "$(date) : file list of dummy test files created."
}
check-dummy-test-files () {
echo "$(date) : test against those test files ..."
file=0
while read i; do
file=$((file+1))
echo -n "$(date) : count $count file: $file ... checking $i ... "
ls -al $i; file $i; md5sum $i
echo ...
##randomise order of dossiers
##done < <(for i in $(cat $HOMEDIR/TEST-GCS-FILES-$TESTFOLDER.txt | perl -pe 's|.*?(tststrng-\w+?)_.*.jpg|$1|' | sort -u | sort -R); do grep $i $HOMEDIR/TEST-GCS-FILES-$TESTFOLDER.txt; done | egrep "original|layout|minihighres|thumbnail|watermark")
#dont randomize order of dossiers
done < <(cat $HOMEDIR/TEST-GCS-FILES-$TESTFOLDER.txt | egrep "original|layout|minihighres|thumbnail|watermark")
echo "$(date) : test of files completed."
}
#main
VERSION="1.04"
SCRIPTBASENAME=$(basename $0)
HOMEDIR=~
LOGFILE=$HOMEDIR/$SCRIPTBASENAME.log.$$
> $LOGFILE
TESTFOLDER="2027"
export fixedstring="tststrng"
MOUNTPOINT="$1"
ACTION="$2"
echo "$(date) : Starting script: $0 Version: $VERSION, MOUNTPOINT: $MOUNTPOINT, ACTION: $ACTION ..."
if [ $# -ne 2 ]; then
usage
else
cat /etc/mtab | grep -q " $MOUNTPOINT " || usage # check MOUNTPOINT exists
case $ACTION in
make-test-files ) make-dummy-test-files
;;
check-test-files ) check-dummy-test-files
;;
make-and-check-test-files ) make-dummy-test-files
check-dummy-test-files
;;
* ) usage
;;
esac | tee -a $LOGFILE
fi
################################################################
System (please complete the following information):
- OS: RHEL 7.9 (Maipo)
- Version gcsfuse/0.41.10
Log file D0118 10:35:12.981651 debug_fs: OpenFile(1642): <nil> D0118 10:35:12.981688 fuse_debug: Op 0x0001b6ed connection.go:498] -> OK () D0118 10:35:12.982286 fuse_debug: Op 0x0001b6ee connection.go:416] <- ReadFile (inode 1642, PID 83008, handle 2493, offset 393216, 131072 bytes) D0118 10:35:12.983113 gcs: Req 0x12f6: <- Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, [393216, 3868762)) D0118 10:35:13.010134 debug_fs: ReadFile(1642, 393216): <nil> D0118 10:35:13.010219 fuse_debug: Op 0x0001b6ee connection.go:498] -> OK () D0118 10:35:13.010505 fuse_debug: Op 0x0001b6ef connection.go:416] <- ReadFile (inode 1642, PID 83008, handle 2493, offset 524288, 131072 bytes) D0118 10:35:13.019444 debug_fs: ReadFile(1642, 524288): <nil> D0118 10:35:13.019504 fuse_debug: Op 0x0001b6ef connection.go:498] -> OK () D0118 10:35:13.019925 fuse_debug: Op 0x0001b6f0 connection.go:416] <- ReadFile (inode 1642, PID 83008, handle 2493, offset 655360, 131072 bytes) D0118 10:37:25.575953 fuse_debug: Op 0x0001b6f1 connection.go:416] <- GetInodeAttributes (inode 1, PID 83450) D0118 10:37:25.576960 debug_fs: GetInodeAttributes(1): <nil> D0118 10:37:25.577001 fuse_debug: Op 0x0001b6f1 connection.go:498] -> OK () D0118 10:37:25.577676 fuse_debug: Op 0x0001b6f2 connection.go:416] <- StatFS D0118 10:37:25.577733 debug_fs: StatFS(): <nil> D0118 10:37:25.577773 fuse_debug: Op 0x0001b6f2 connection.go:498] -> OK () D0118 10:37:26.638600 fuse_debug: Op 0x0001b6f3 connection.go:416] <- StatFS D0118 10:37:26.638709 debug_fs: StatFS(): <nil> D0118 10:37:26.638751 fuse_debug: Op 0x0001b6f3 connection.go:498] -> OK () D0118 10:39:07.780974 fuse_debug: Op 0x0001b6f4 connection.go:416] <- GetInodeAttributes (inode 1, PID 83842) D0118 10:39:07.782106 debug_fs: GetInodeAttributes(1): <nil> D0118 10:39:07.782171 fuse_debug: Op 0x0001b6f4 connection.go:498] -> OK () D0118 10:39:07.782876 fuse_debug: Op 0x0001b6f5 connection.go:416] <- StatFS D0118 10:39:07.782920 debug_fs: StatFS(): <nil> D0118 10:39:07.782949 fuse_debug: Op 0x0001b6f5 connection.go:498] -> OK () D0118 10:39:56.912562 fuse_debug: Op 0x0001b6f6 connection.go:416] <- StatFS D0118 10:39:56.912645 debug_fs: StatFS(): <nil> D0118 10:39:56.912717 fuse_debug: Op 0x0001b6f6 connection.go:498] -> OK () I0118 10:41:12.672368 Starting a garbage collection run. D0118 10:41:12.672479 gcs: Req 0x12f7: <- ListObjects(“.gcsfuse_tmp/”) D0118 10:41:13.042569 gcs: Req 0x12f6: -> Read error: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): stream error: stream ID 9709; INTERNAL_ERROR; received from peer D0118 10:41:13.042632 debug_fs: ReadFile(1642, 655360): fh.reader.ReadAt: readFull: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): stream error: stream ID 9709; INTERNAL_ERROR; received from peer E0118 10:41:13.042654 ReadFile: input/output error, fh.reader.ReadAt: readFull: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): stream error: stream ID 9709; INTERNAL_ERROR; received from peer D0118 10:41:13.042921 fuse_debug: Op 0x0001b6f0 connection.go:500] -> Error: “input/output error” E0118 10:41:13.042943 fuse: *fuseops.ReadFileOp error: input/output error D0118 10:41:13.043181 fuse_debug: Op 0x0001b6f7 connection.go:416] <- ReadFile (inode 1642, PID 83008, handle 2493, offset 786432, 131072 bytes) D0118 10:41:13.044438 gcs: Req 0x12f6: -> Read error: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): stream error: stream ID 9709; INTERNAL_ERROR; received from peer D0118 10:41:13.044479 gcs: Req 0x12f6: -> Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, [393216, 3868762)) (6m0.061381102s): OK D0118 10:41:13.044649 gcs: Req 0x12f8: <- Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, [786432, 3868762)) D0118 10:42:27.142512 fuse_debug: Op 0x0001b6f8 connection.go:416] <- GetInodeAttributes (inode 1, PID 84787) D0118 10:42:27.143570 debug_fs: GetInodeAttributes(1): <nil> D0118 10:42:27.143634 fuse_debug: Op 0x0001b6f8 connection.go:498] -> OK () D0118 10:42:27.144486 fuse_debug: Op 0x0001b6f9 connection.go:416] <- StatFS D0118 10:42:27.144614 debug_fs: StatFS(): <nil> D0118 10:42:27.144674 fuse_debug: Op 0x0001b6f9 connection.go:498] -> OK () D0118 10:44:56.293259 fuse_debug: Op 0x0001b6fa connection.go:416] <- GetInodeAttributes (inode 1, PID 85255) D0118 10:44:56.294280 debug_fs: GetInodeAttributes(1): <nil> D0118 10:44:56.294322 fuse_debug: Op 0x0001b6fa connection.go:498] -> OK () D0118 10:44:56.295211 fuse_debug: Op 0x0001b6fb connection.go:416] <- StatFS D0118 10:44:56.295274 debug_fs: StatFS(): <nil> D0118 10:44:56.295319 fuse_debug: Op 0x0001b6fb connection.go:498] -> OK () D0118 10:47:12.734145 gcs: Req 0x12f7: -> ListObjects(“.gcsfuse_tmp/”) (6m0.061658879s): not retrying ListObjects(“.gcsfuse_tmp/”): Get “https://storage.googleapis.com:443/storage/v1/b/ph-dev-images-bkt/o?prefix=.gcsfuse_tmp%2F&projection=full”: stream error: stream ID 9711; INTERNAL_ERROR; received from peer I0118 10:47:12.734217 Garbage collection failed after deleting 0 objects in 6m0.061790819s, with error: ListPrefix: ListObjects: not retrying ListObjects(“.gcsfuse_tmp/”): Get “https://storage.googleapis.com:443/storage/v1/b/ph-dev-images-bkt/o?prefix=.gcsfuse_tmp%2F&projection=full”: stream error: stream ID 9711; INTERNAL_ERROR; received from peer D0118 10:47:13.103065 gcs: Req 0x12f8: -> Read error: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): Get “https://storage.googleapis.com:443/download/storage/v1/b/ph-dev-images-bkt/o/device%2Fimagearc%2F2027%2F01-08%2Fca%2Fa7%2F63%2Ftststrng-kkvps7frsz2cuxy58lha_original.jpg?alt=media&generation=1673924215857402”: stream error: stream ID 9713; INTERNAL_ERROR; received from peer D0118 10:47:13.103128 debug_fs: ReadFile(1642, 786432): fh.reader.ReadAt: readFull: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): Get “https://storage.googleapis.com:443/download/storage/v1/b/ph-dev-images-bkt/o/device%2Fimagearc%2F2027%2F01-08%2Fca%2Fa7%2F63%2Ftststrng-kkvps7frsz2cuxy58lha_original.jpg?alt=media&generation=1673924215857402”: stream error: stream ID 9713; INTERNAL_ERROR; received from peer E0118 10:47:13.103153 ReadFile: input/output error, fh.reader.ReadAt: readFull: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): Get “https://storage.googleapis.com:443/download/storage/v1/b/ph-dev-images-bkt/o/device%2Fimagearc%2F2027%2F01-08%2Fca%2Fa7%2F63%2Ftststrng-kkvps7frsz2cuxy58lha_original.jpg?alt=media&generation=1673924215857402”: stream error: stream ID 9713; INTERNAL_ERROR; received from peer D0118 10:47:13.103195 fuse_debug: Op 0x0001b6f7 connection.go:500] -> Error: “input/output error” E0118 10:47:13.103207 fuse: *fuseops.ReadFileOp error: input/output error D0118 10:47:13.104055 fuse_debug: Op 0x0001b6fc connection.go:416] <- ReadFile (inode 1642, PID 83008, handle 2493, offset 917504, 131072 bytes) D0118 10:47:13.105233 gcs: Req 0x12f8: -> Read error: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): Get “https://storage.googleapis.com:443/download/storage/v1/b/ph-dev-images-bkt/o/device%2Fimagearc%2F2027%2F01-08%2Fca%2Fa7%2F63%2Ftststrng-kkvps7frsz2cuxy58lha_original.jpg?alt=media&generation=1673924215857402”: stream error: stream ID 9713; INTERNAL_ERROR; received from peer D0118 10:47:13.105266 gcs: Req 0x12f8: -> Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, [786432, 3868762)) (6m0.060644011s): OK D0118 10:47:13.105301 gcs: Req 0x12f9: <- Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, [917504, 1966080)) D0118 10:47:26.445422 fuse_debug: Op 0x0001b6fd connection.go:416] <- GetInodeAttributes (inode 1, PID 85772) D0118 10:47:26.446392 debug_fs: GetInodeAttributes(1): <nil> D0118 10:47:26.446439 fuse_debug: Op 0x0001b6fd connection.go:498] -> OK () D0118 10:47:26.447473 fuse_debug: Op 0x0001b6fe connection.go:416] <- StatFS D0118 10:47:26.447542 debug_fs: StatFS(): <nil> D0118 10:47:26.447572 fuse_debug: Op 0x0001b6fe connection.go:498] -> OK () D0118 10:49:56.697244 fuse_debug: Op 0x0001b6ff connection.go:416] <- GetInodeAttributes (inode 1, PID 86275) D0118 10:49:56.698634 debug_fs: GetInodeAttributes(1): <nil> D0118 10:49:56.698694 fuse_debug: Op 0x0001b6ff connection.go:498] -> OK () D0118 10:49:56.699604 fuse_debug: Op 0x0001b700 connection.go:416] <- StatFS D0118 10:49:56.699740 debug_fs: StatFS(): <nil> D0118 10:49:56.699776 fuse_debug: Op 0x0001b700 connection.go:498] -> OK () I0118 10:51:12.671879 Starting a garbage collection run. D0118 10:51:12.671978 gcs: Req 0x12fa: <- ListObjects(“.gcsfuse_tmp/”) D0118 10:52:26.880878 fuse_debug: Op 0x0001b701 connection.go:416] <- GetInodeAttributes (inode 1, PID 87063) D0118 10:52:26.881986 debug_fs: GetInodeAttributes(1): <nil> D0118 10:52:26.882029 fuse_debug: Op 0x0001b701 connection.go:498] -> OK () D0118 10:52:26.882943 fuse_debug: Op 0x0001b702 connection.go:416] <- StatFS D0118 10:52:26.883037 debug_fs: StatFS(): <nil> D0118 10:52:26.883067 fuse_debug: Op 0x0001b702 connection.go:498] -> OK () D0118 10:53:13.168672 gcs: Req 0x12f9: -> Read error: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): stream error: stream ID 9715; INTERNAL_ERROR; received from peer D0118 10:53:13.168733 debug_fs: ReadFile(1642, 917504): fh.reader.ReadAt: readFull: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): stream error: stream ID 9715; INTERNAL_ERROR; received from peer E0118 10:53:13.168762 ReadFile: input/output error, fh.reader.ReadAt: readFull: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): stream error: stream ID 9715; INTERNAL_ERROR; received from peer D0118 10:53:13.168807 fuse_debug: Op 0x0001b6fc connection.go:500] -> Error: “input/output error” E0118 10:53:13.168819 fuse: *fuseops.ReadFileOp error: input/output error D0118 10:53:13.169006 fuse_debug: Op 0x0001b703 connection.go:416] <- ReadFile (inode 1642, PID 83008, handle 2493, offset 1048576, 131072 bytes) D0118 10:53:13.170223 gcs: Req 0x12f9: -> Read error: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): stream error: stream ID 9715; INTERNAL_ERROR; received from peer D0118 10:53:13.170267 gcs: Req 0x12f9: -> Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, [917504, 1966080)) (6m0.064977234s): OK D0118 10:53:13.170307 gcs: Req 0x12fb: <- Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, [1048576, 2097152)) D0118 10:54:08.027912 fuse_debug: Op 0x0001b704 connection.go:416] <- GetInodeAttributes (inode 1, PID 87427) D0118 10:54:08.029075 debug_fs: GetInodeAttributes(1): <nil> D0118 10:54:08.029328 fuse_debug: Op 0x0001b704 connection.go:498] -> OK () D0118 10:54:08.030145 fuse_debug: Op 0x0001b705 connection.go:416] <- StatFS D0118 10:54:08.030201 debug_fs: StatFS(): <nil> D0118 10:54:08.030247 fuse_debug: Op 0x0001b705 connection.go:498] -> OK () D0118 10:54:57.134910 fuse_debug: Op 0x0001b706 connection.go:416] <- StatFS D0118 10:54:57.134993 debug_fs: StatFS(): <nil> D0118 10:54:57.135042 fuse_debug: Op 0x0001b706 connection.go:498] -> OK () D0118 10:57:12.729207 gcs: Req 0x12fa: -> ListObjects(“.gcsfuse_tmp/”) (6m0.05722955s): not retrying ListObjects(“.gcsfuse_tmp/”): Get “https://storage.googleapis.com:443/storage/v1/b/ph-dev-images-bkt/o?prefix=.gcsfuse_tmp%2F&projection=full”: stream error: stream ID 9717; INTERNAL_ERROR; received from peer I0118 10:57:12.729271 Garbage collection failed after deleting 0 objects in 6m0.0573484s, with error: ListPrefix: ListObjects: not retrying ListObjects(“.gcsfuse_tmp/”): Get “https://storage.googleapis.com:443/storage/v1/b/ph-dev-images-bkt/o?prefix=.gcsfuse_tmp%2F&projection=full”: stream error: stream ID 9717; INTERNAL_ERROR; received from peer D0118 10:57:26.329697 fuse_debug: Op 0x0001b707 connection.go:416] <- GetInodeAttributes (inode 1, PID 88085) D0118 10:57:26.331439 debug_fs: GetInodeAttributes(1): <nil> D0118 10:57:26.331510 fuse_debug: Op 0x0001b707 connection.go:498] -> OK () D0118 10:57:26.333019 fuse_debug: Op 0x0001b708 connection.go:416] <- StatFS D0118 10:57:26.333074 debug_fs: StatFS(): <nil> D0118 10:57:26.333117 fuse_debug: Op 0x0001b708 connection.go:498] -> OK () D0118 10:59:13.232178 gcs: Req 0x12fb: -> Read error: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): Get “https://storage.googleapis.com:443/download/storage/v1/b/ph-dev-images-bkt/o/device%2Fimagearc%2F2027%2F01-08%2Fca%2Fa7%2F63%2Ftststrng-kkvps7frsz2cuxy58lha_original.jpg?alt=media&generation=1673924215857402”: stream error: stream ID 9719; INTERNAL_ERROR; received from peer D0118 10:59:13.232258 debug_fs: ReadFile(1642, 1048576): fh.reader.ReadAt: readFull: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): Get “https://storage.googleapis.com:443/download/storage/v1/b/ph-dev-images-bkt/o/device%2Fimagearc%2F2027%2F01-08%2Fca%2Fa7%2F63%2Ftststrng-kkvps7frsz2cuxy58lha_original.jpg?alt=media&generation=1673924215857402”: stream error: stream ID 9719; INTERNAL_ERROR; received from peer E0118 10:59:13.232327 ReadFile: input/output error, fh.reader.ReadAt: readFull: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): Get “https://storage.googleapis.com:443/download/storage/v1/b/ph-dev-images-bkt/o/device%2Fimagearc%2F2027%2F01-08%2Fca%2Fa7%2F63%2Ftststrng-kkvps7frsz2cuxy58lha_original.jpg?alt=media&generation=1673924215857402”: stream error: stream ID 9719; INTERNAL_ERROR; received from peer D0118 10:59:13.232654 fuse_debug: Op 0x0001b703 connection.go:500] -> Error: “input/output error” E0118 10:59:13.232672 fuse: *fuseops.ReadFileOp error: input/output error D0118 10:59:13.232802 fuse_debug: Op 0x0001b709 connection.go:416] <- ReadFile (inode 1642, PID 83008, handle 2493, offset 655360, 4096 bytes) D0118 10:59:13.233891 gcs: Req 0x12fb: -> Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, [1048576, 2097152)) (6m0.063602749s): OK D0118 10:59:13.233953 gcs: Req 0x12fc: <- Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, [655360, 1703936)) D0118 10:59:56.571908 fuse_debug: Op 0x0001b70a connection.go:416] <- GetInodeAttributes (inode 1, PID 88589) D0118 10:59:56.573033 debug_fs: GetInodeAttributes(1): <nil> D0118 10:59:56.573079 fuse_debug: Op 0x0001b70a connection.go:498] -> OK () D0118 10:59:56.574020 fuse_debug: Op 0x0001b70b connection.go:416] <- StatFS D0118 10:59:56.574084 debug_fs: StatFS(): <nil> D0118 10:59:56.574121 fuse_debug: Op 0x0001b70b connection.go:498] -> OK () D0118 11:00:02.143380 fuse_debug: Op 0x0001b70c connection.go:416] <- StatFS D0118 11:00:02.145980 debug_fs: StatFS(): <nil> D0118 11:00:02.146110 fuse_debug: Op 0x0001b70c connection.go:498] -> OK () D0118 11:00:03.265753 fuse_debug: Op 0x0001b70d connection.go:416] <- OpenDir (inode 1, PID 89622) D0118 11:00:03.266980 debug_fs: OpenDir(1): <nil> D0118 11:00:03.267035 fuse_debug: Op 0x0001b70d connection.go:498] -> OK () D0118 11:00:03.267508 fuse_debug: Op 0x0001b70e connection.go:416] <- ReleaseDirHandle (PID 0) D0118 11:00:03.267550 fuse_debug: Op 0x0001b70f connection.go:416] <- StatFS D0118 11:00:03.268396 debug_fs: StatFS(): <nil> D0118 11:00:03.268470 fuse_debug: Op 0x0001b70f connection.go:498] -> OK () D0118 11:00:03.269638 debug_fs: ReleaseDirHandle(2494): <nil> D0118 11:00:03.269690 fuse_debug: Op 0x0001b70e connection.go:498] -> OK () D0118 11:00:03.344193 fuse_debug: Op 0x0001b710 connection.go:416] <- StatFS D0118 11:00:03.344268 debug_fs: StatFS(): <nil> D0118 11:00:03.344323 fuse_debug: Op 0x0001b710 connection.go:498] -> OK () D0118 11:01:09.373163 fuse_debug: Op 0x0001b711 connection.go:416] <- GetInodeAttributes (inode 1, PID 89905) D0118 11:01:09.374236 debug_fs: GetInodeAttributes(1): <nil> D0118 11:01:09.374294 fuse_debug: Op 0x0001b711 connection.go:498] -> OK () D0118 11:01:09.374463 fuse_debug: Op 0x0001b712 connection.go:416] <- OpenDir (inode 1, PID 89905) D0118 11:01:09.375330 debug_fs: OpenDir(1): <nil> D0118 11:01:09.375386 fuse_debug: Op 0x0001b712 connection.go:498] -> OK () D0118 11:01:09.376295 fuse_debug: Op 0x0001b713 connection.go:416] <- ReleaseDirHandle (PID 0) D0118 11:01:09.376348 fuse_debug: Op 0x0001b714 connection.go:416] <- StatFS D0118 11:01:09.377209 debug_fs: ReleaseDirHandle(2495): <nil> D0118 11:01:09.377255 fuse_debug: Op 0x0001b713 connection.go:498] -> OK () D0118 11:01:09.377278 debug_fs: StatFS(): <nil> D0118 11:01:09.377515 fuse_debug: Op 0x0001b714 connection.go:498] -> OK () I0118 11:01:12.672662 Starting a garbage collection run. D0118 11:01:12.672806 gcs: Req 0x12fd: <- ListObjects(“.gcsfuse_tmp/”) D0118 11:01:19.058072 fuse_debug: Op 0x0001b715 connection.go:416] <- OpenDir (inode 1, PID 89915) D0118 11:01:19.059111 debug_fs: OpenDir(1): <nil> D0118 11:01:19.059168 fuse_debug: Op 0x0001b715 connection.go:498] -> OK () D0118 11:01:19.059313 fuse_debug: Op 0x0001b716 connection.go:416] <- ReadDir (inode 1, PID 89915) D0118 11:01:19.060325 gcs: Req 0x12fe: <- ListObjects(“”) D0118 11:01:24.060484 debug_mutexPotential dead lock detected for a lock “” held by: goroutine 220099 [running]: github.com/googlecloudplatform/gcsfuse/internal/locker.(*debugger).Lock(0xc00008de80) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/locker/locker.go:93 +0x59 github.com/googlecloudplatform/gcsfuse/internal/fs/inode.(*dirInode).Lock(0x0?) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/inode/dir.go:368 +0x28 github.com/googlecloudplatform/gcsfuse/internal/fs.(*dirHandle).ensureEntries(0xc00087e2d0, {0x109def0, 0xc0014e3530}) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/dir_handle.go:224 +0x57 github.com/googlecloudplatform/gcsfuse/internal/fs.(*dirHandle).ReadDir(0xc00087e2d0, {0x109def0?, 0xc0014e3530?}, 0xc001026700) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/dir_handle.go:266 +0x8e github.com/googlecloudplatform/gcsfuse/internal/fs.(*fileSystem).ReadDir(0xc000264300, {0x109def0, 0xc0014e3530}, 0xc001026700) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/fs.go:1689 +0x127 github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*debugLogging).ReadDir(0xc0002284f8, {0x109def0?, 0xc0014e3530?}, 0xc001026700) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/debug_logging.go:167 +0x4b github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*errorMapping).ReadDir(0xc000228558, {0x109def0?, 0xc0014e3530?}, 0x0?) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/error_mapping.go:213 +0x36 github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*monitoring).ReadDir(0xc00003d070, {0x109def0, 0xc0014e3530}, 0xc0014e77e0?) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/monitoring.go:287 +0x79 github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).handleOp(0x441 D0118 11:01:24.060554 debug_mutexPotential dead lock detected for a lock “DH.” held by: goroutine 220099 [running]: github.com/googlecloudplatform/gcsfuse/internal/locker.(*debugger).Lock(0xc0005e4340) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/locker/locker.go:93 +0x59 github.com/googlecloudplatform/gcsfuse/internal/fs.(*fileSystem).ReadDir(0xc000264300, {0x109def0, 0xc0014e3530}, 0xc001026700) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/fs.go:1685 +0xca github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*debugLogging).ReadDir(0xc0002284f8, {0x109def0?, 0xc0014e3530?}, 0xc001026700) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/debug_logging.go:167 +0x4b github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*errorMapping).ReadDir(0xc000228558, {0x109def0?, 0xc0014e3530?}, 0x0?) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/error_mapping.go:213 +0x36 github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*monitoring).ReadDir(0xc00003d070, {0x109def0, 0xc0014e3530}, 0xc0014e77e0?) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/monitoring.go:287 +0x79 github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).handleOp(0x441045?, 0xc000033200?, {0x109def0, 0xc0014e3530}, {0xc6b8e0?, 0xc001026700?}) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil/file_system.go:199 +0xa83 created by github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).ServeOps /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil/file_system.go:123 +0x205
D0118 11:02:26.823406 fuse_debug: Op 0x0001b717 connection.go:416] <- GetInodeAttributes (inode 1, PID 90187) D0118 11:02:28.795791 fuse_debug: Op 0x0001b718 connection.go:416] <- interrupt (fuseid 0x0001b717) D0118 11:02:33.799298 fuse_debug: Op 0x0001b719 connection.go:416] <- GetInodeAttributes (inode 1, PID 90212) D0118 11:02:34.455941 fuse_debug: Op 0x0001b71a connection.go:416] <- interrupt (fuseid 0x0001b716) D0118 11:02:34.456122 gcs: Req 0x12fe: -> ListObjects(“”) (1m15.39579473s): not retrying ListObjects(“”): Get “https://storage.googleapis.com:443/storage/v1/b/ph-dev-images-bkt/o?delimiter=%2F&includeTrailingDelimiter=true&maxResults=5000&projection=noAcl”: net/http: request canceled D0118 11:02:34.456161 debug_fs: ReadDir(1, 0): readAllEntries: ReadEntries: read objects: ListObjects: not retrying ListObjects(“”): Get “https://storage.googleapis.com:443/storage/v1/b/ph-dev-images-bkt/o?delimiter=%2F&includeTrailingDelimiter=true&maxResults=5000&projection=noAcl”: net/http: request canceled E0118 11:02:34.456176 ReadDir: operation canceled, readAllEntries: ReadEntries: read objects: ListObjects: not retrying ListObjects(“”): Get “https://storage.googleapis.com:443/storage/v1/b/ph-dev-images-bkt/o?delimiter=%2F&includeTrailingDelimiter=true&maxResults=5000&projection=noAcl”: net/http: request canceled D0118 11:02:34.456219 fuse_debug: Op 0x0001b716 connection.go:500] -> Error: “operation canceled” E0118 11:02:34.456230 fuse: *fuseops.ReadDirOp error: operation canceled D0118 11:02:34.456291 debug_fs: GetInodeAttributes(1): <nil> D0118 11:02:34.456311 fuse_debug: Op 0x0001b717 connection.go:498] -> OK () D0118 11:02:34.456341 debug_fs: GetInodeAttributes(1): <nil> D0118 11:02:34.456357 fuse_debug: Op 0x0001b719 connection.go:498] -> OK () D0118 11:02:34.456975 fuse_debug: Op 0x0001b71b connection.go:416] <- ReleaseDirHandle (PID 0) D0118 11:02:34.457561 fuse_debug: Op 0x0001b71c connection.go:416] <- StatFS D0118 11:02:34.457843 debug_fs: StatFS(): <nil> D0118 11:02:34.457905 fuse_debug: Op 0x0001b71c connection.go:498] -> OK () D0118 11:02:34.458273 debug_fs: ReleaseDirHandle(2496): <nil> D0118 11:02:34.458316 fuse_debug: Op 0x0001b71b connection.go:498] -> OK () D0118 11:02:37.327984 fuse_debug: Op 0x0001b71d connection.go:416] <- OpenDir (inode 1, PID 90222) D0118 11:02:37.329199 debug_fs: OpenDir(1): <nil> D0118 11:02:37.329276 fuse_debug: Op 0x0001b71d connection.go:498] -> OK () D0118 11:02:37.329678 fuse_debug: Op 0x0001b71e connection.go:416] <- ReadDir (inode 1, PID 90222) D0118 11:02:37.330416 gcs: Req 0x12ff: <- ListObjects(“”) D0118 11:02:42.330588 debug_mutexPotential dead lock detected for a lock “” held by: goroutine 220117 [running]: github.com/googlecloudplatform/gcsfuse/internal/locker.(*debugger).Lock(0xc00008de80) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/locker/locker.go:93 +0x59 github.com/googlecloudplatform/gcsfuse/internal/fs/inode.(*dirInode).Lock(0x0?) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/inode/dir.go:368 +0x28 github.com/googlecloudplatform/gcsfuse/internal/fs.(*dirHandle).ensureEntries(0xc001119f40, {0x109def0, 0xc0015aeab0}) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/dir_handle.go:224 +0x57 github.com/googlecloudplatform/gcsfuse/internal/fs.(*dirHandle).ReadDir(0xc001119f40, {0x109def0?, 0xc0015aeab0?}, 0xc0005e4f40) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/dir_handle.go:266 +0x8e github.com/googlecloudplatform/gcsfuse/internal/fs.(*fileSystem).ReadDir(0xc000264300, {0x109def0, 0xc0015aeab0}, 0xc0005e4f40) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/fs.go:1689 +0x127 github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*debugLogging).ReadDir(0xc0002284f8, {0x109def0?, 0xc0015aeab0?}, 0xc0005e4f40) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/debug_logging.go:167 +0x4b github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*errorMapping).ReadDir(0xc000228558, {0x109def0?, 0xc0015aeab0?}, 0x0?) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/error_mapping.go:213 +0x36 github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*monitoring).ReadDir(0xc00003d070, {0x109def0, 0xc0015aeab0}, 0xc00116a1a0?) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/monitoring.go:287 +0x79 github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).handleOp(0xc00 D0118 11:02:42.330657 debug_mutexPotential dead lock detected for a lock “DH.” held by: goroutine 220117 [running]: github.com/googlecloudplatform/gcsfuse/internal/locker.(*debugger).Lock(0xc001026d00) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/locker/locker.go:93 +0x59 github.com/googlecloudplatform/gcsfuse/internal/fs.(*fileSystem).ReadDir(0xc000264300, {0x109def0, 0xc0015aeab0}, 0xc0005e4f40) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/fs.go:1685 +0xca github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*debugLogging).ReadDir(0xc0002284f8, {0x109def0?, 0xc0015aeab0?}, 0xc0005e4f40) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/debug_logging.go:167 +0x4b github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*errorMapping).ReadDir(0xc000228558, {0x109def0?, 0xc0015aeab0?}, 0x0?) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/error_mapping.go:213 +0x36 github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*monitoring).ReadDir(0xc00003d070, {0x109def0, 0xc0015aeab0}, 0xc00116a1a0?) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/monitoring.go:287 +0x79 github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).handleOp(0xc000c5a838?, 0xc00008de80?, {0x109def0, 0xc0015aeab0}, {0xc6b8e0?, 0xc0005e4f40?}) /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil/file_system.go:199 +0xa83 created by github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).ServeOps /tmp/build_gcsfuse_gopath2289345396/src/github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil/file_system.go:123 +0x205
D0118 11:03:19.286615 fuse_debug: Op 0x0001b71f connection.go:416] <- interrupt (fuseid 0x0001b71e) D0118 11:03:19.286781 gcs: Req 0x12ff: -> ListObjects(“”) (41.956359859s): not retrying ListObjects(“”): Get “https://storage.googleapis.com:443/storage/v1/b/ph-dev-images-bkt/o?delimiter=%2F&includeTrailingDelimiter=true&maxResults=5000&projection=noAcl”: net/http: request canceled D0118 11:03:19.286831 debug_fs: ReadDir(1, 0): readAllEntries: ReadEntries: read objects: ListObjects: not retrying ListObjects(“”): Get “https://storage.googleapis.com:443/storage/v1/b/ph-dev-images-bkt/o?delimiter=%2F&includeTrailingDelimiter=true&maxResults=5000&projection=noAcl”: net/http: request canceled E0118 11:03:19.286850 ReadDir: operation canceled, readAllEntries: ReadEntries: read objects: ListObjects: not retrying ListObjects(“”): Get “https://storage.googleapis.com:443/storage/v1/b/ph-dev-images-bkt/o?delimiter=%2F&includeTrailingDelimiter=true&maxResults=5000&projection=noAcl”: net/http: request canceled D0118 11:03:19.286897 fuse_debug: Op 0x0001b71e connection.go:500] -> Error: “operation canceled” E0118 11:03:19.286910 fuse: *fuseops.ReadDirOp error: operation canceled D0118 11:03:19.288905 fuse_debug: Op 0x0001b720 connection.go:416] <- ReleaseDirHandle (PID 0) D0118 11:03:19.289933 debug_fs: ReleaseDirHandle(2497): <nil> D0118 11:03:19.289995 fuse_debug: Op 0x0001b720 connection.go:498] -> OK () D0118 11:04:57.065843 fuse_debug: Op 0x0001b721 connection.go:416] <- GetInodeAttributes (inode 1, PID 90682) D0118 11:04:57.067004 debug_fs: GetInodeAttributes(1): <nil> D0118 11:04:57.067059 fuse_debug: Op 0x0001b721 connection.go:498] -> OK () D0118 11:04:57.067924 fuse_debug: Op 0x0001b722 connection.go:416] <- StatFS D0118 11:04:57.068040 debug_fs: StatFS(): <nil> D0118 11:04:57.068088 fuse_debug: Op 0x0001b722 connection.go:498] -> OK () D0118 11:05:13.298775 gcs: Req 0x12fc: -> Read error: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): stream error: stream ID 9721; INTERNAL_ERROR; received from peer D0118 11:05:13.298855 debug_fs: ReadFile(1642, 655360): fh.reader.ReadAt: readFull: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): stream error: stream ID 9721; INTERNAL_ERROR; received from peer E0118 11:05:13.298898 ReadFile: input/output error, fh.reader.ReadAt: readFull: not retrying Read(“device/imagearc/2027/01-08/ca/a7/63/tststrng-kkvps7frsz2cuxy58lha_original.jpg”, 1673924215857402): stream error: stream ID 9721; INTERNAL_ERROR; received from peer D0118 11:05:13.298985 fuse_debug: Op 0x0001b709 connection.go:500] -> Error: “input/output error” E0118 11:05:13.299004 fuse: *fuseops.ReadFileOp error: input/output error D0118 11:05:13.299308 fuse_debug: Op 0x0001b723 connection.go:416] <- FlushFile (inode 1642, PID 83008) D0118 11:05:13.300774 debug_fs: FlushFile(1642): <nil>
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 19
Hii @dhazell, you can use -disable_http2 flag. Meanwhile, we are looking into Input/output errors. Can you please share GCSFuse logs in this case ? (–debug_gcs, --debug_fs, --debug_fuse) ?