gcsfuse: internal error causing ls command to show input/output error in k8s pod
Describe the issue Please provide a clear description of what you were trying to achieve along with the details of the flags that you passed.
I was able to use gcsfuse to access all my objects in GCS bucket’s subdir in my container with a gcsfuse sidercar, and run some python code with pytorch over the objects in GCS bucket to train a model until suddenly halfway through training the model alot error starts to show up saying nothing found in filesystem.
System & Version (please complete the following information):
- OS: Ubuntu in docker
- Platform GKE Autopilot using CSI driver to mount on GCS bucket with sidecar
- Version
- Gcsfuse sidecar mounter version v1.4.3-gke.1, GKE version
- Client Version: v1.29.7
- Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
- Server Version: v1.29.7-gke.1008000
Steps to reproduce the behavior with following information:
- Please share Mount command including all command line or config flags used to mount the bucket.
- did it throw the following kopf operator code to create deployment
"volumes": [{
"name": "data-volume",
"csi": {
"driver": "gcsfuse.csi.storage.gke.io",
"volumeAttributes": {
"bucketName": spec['bucket_name'],
"mountOptions": f"implicit-dirs,only-dir={name}/,uid=1000,gid=100,file-mode=0777,dir-mode=0777"
}
}
}]
- Please make sure you have no other security, monitoring, background processes which can offend the FUSE process running. Possibly reproduce under a fresh/clean installation.
- Please rerun with --debug_fuse --debug_fs --debug_gcs --debug_http --foreground as additional flags to enable debug logs. I dont know how to run these in the sidecar, but tried
% kubectl exec -it c0a16cff-473b-44d8-8217-de2028e29bb4-5cbc9646c-wxzwt -c gke-gcsfuse-sidecar -- /bin/bash
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "5e4b60642a54f33f41ffdbce79fd09a9705192499ff1fa88eb81b2bc60c04c03": OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown
- Monitor the logs and please capture screenshots or copy the relevant logs to a file (can use --log-format and --log-file as well). gcsfuse-sidercar container log
I0823 04:06:52.299238 1 main.go:48] Running Google Cloud Storage FUSE CSI driver sidecar mounter version v1.4.3-gke.1
I0823 04:06:53.800146 1 sidecar_mounter_config.go:101] connecting to socket "/gcsfuse-tmp/.volumes/data-volume/socket"
I0823 04:06:53.800750 1 fdchannel.go:48] get the underlying socket
I0823 04:06:53.800771 1 fdchannel.go:60] calling recvmsg...
I0823 04:06:53.800984 1 fdchannel.go:69] parsing SCM...
I0823 04:06:53.800994 1 fdchannel.go:76] parsing SCM_RIGHTS...
I0823 04:06:53.801194 1 sidecar_mounter_config.go:269] gcsfuse config file content: map[cache-dir: logging:map[file-path:/dev/fd/1 format:json]]
I0823 04:06:53.801280 1 sidecar_mounter.go:51] start to mount bucket "planetgpt" for volume "data-volume"
I0823 04:06:53.801396 1 sidecar_mounter.go:70] gcsfuse mounting with args [--app-name gke-gcs-fuse-csi --gid 100 --config-file /gcsfuse-tmp/.volumes/data-volume/config.yaml --dir-mode 0777 --only-dir c0a16cff-473b-44d8-8217-de2028e29bb4/ --foreground --uid 1000 --temp-dir /gcsfuse-buffer/.volumes/data-volume/temp-dir --file-mode 0777 --implicit-dirs planetgpt /dev/fd/3]...
I0823 04:06:53.801535 1 main.go:73] waiting for SIGTERM signal...
I0823 04:06:53.803078 1 sidecar_mounter.go:105] gcsfuse for bucket "planetgpt", volume "data-volume" started with process id 17
{"timestamp":{"seconds":1724386013,"nanos":887543056},"severity":"INFO","message":"Start gcsfuse/2.3.2-gke.0 (Go version go1.22.4) for app \"gke-gcs-fuse-csi\" using mount point: /dev/fd/3\n"}
{"timestamp":{"seconds":1724386013,"nanos":887865368},"severity":"INFO","message":"GCSFuse mount command flags: {\"AppName\":\"gke-gcs-fuse-csi\",\"Foreground\":true,\"ConfigFile\":\"/gcsfuse-tmp/.volumes/data-volume/config.yaml\",\"MountOptions\":{},\"DirMode\":511,\"FileMode\":511,\"Uid\":1000,\"Gid\":100,\"ImplicitDirs\":true,\"OnlyDir\":\"subdir1/\",\"RenameDirLimit\":0,\"IgnoreInterrupts\":true,\"CustomEndpoint\":null,\"BillingProject\":\"\",\"KeyFile\":\"\",\"TokenUrl\":\"\",\"ReuseTokenFromUrl\":true,\"EgressBandwidthLimitBytesPerSecond\":-1,\"OpRateLimitHz\":-1,\"SequentialReadSizeMb\":200,\"AnonymousAccess\":false,\"MaxRetrySleep\":30000000000,\"StatCacheCapacity\":20460,\"StatCacheTTL\":60000000000,\"TypeCacheTTL\":60000000000,\"KernelListCacheTtlSeconds\":0,\"HttpClientTimeout\":0,\"MaxRetryDuration\":-1000000000,\"RetryMultiplier\":2,\"LocalFileCache\":false,\"TempDir\":\"/gcsfuse-buffer/.volumes/data-volume/temp-dir\",\"ClientProtocol\":\"http1\",\"MaxConnsPerHost\":0,\"MaxIdleConnsPerHost\":100,\"EnableNonexistentTypeCache\":false,\"StackdriverExportInterval\":0,\"OtelCollectorAddress\":\"\",\"LogFile\":\"\",\"LogFormat\":\"json\",\"ExperimentalEnableJsonRead\":false,\"DebugFuseErrors\":true,\"DebugFuse\":false,\"DebugFS\":false,\"DebugGCS\":false,\"DebugHTTP\":false,\"DebugInvariants\":false,\"DebugMutex\":false,\"ExperimentalMetadataPrefetchOnMount\":\"disabled\"}"}
{"timestamp":{"seconds":1724386013,"nanos":887965999},"severity":"INFO","message":"GCSFuse mount config flags: {\"CreateEmptyFile\":false,\"Severity\":\"INFO\",\"Format\":\"json\",\"FilePath\":\"/dev/fd/1\",\"LogRotateConfig\":{\"MaxFileSizeMB\":512,\"BackupFileCount\":10,\"Compress\":true},\"MaxSizeMB\":-1,\"CacheFileForRangeRead\":false,\"EnableParallelDownloads\":false,\"ParallelDownloadsPerFile\":16,\"MaxParallelDownloads\":96,\"DownloadChunkSizeMB\":50,\"EnableCRC\":false,\"CacheDir\":\"\",\"TtlInSeconds\":-9223372036854775808,\"TypeCacheMaxSizeMB\":4,\"StatCacheMaxSizeMB\":-9223372036854775808,\"EnableEmptyManagedFolders\":false,\"GRPCConnPoolSize\":1,\"AnonymousAccess\":false,\"EnableHNS\":false,\"IgnoreInterrupts\":true,\"DisableParallelDirops\":false,\"KernelListCacheTtlSeconds\":0}"}
{"timestamp":{"seconds":1724386013,"nanos":888011117},"severity":"INFO","message":"Creating Storage handle..."}
{"timestamp":{"seconds":1724386013,"nanos":888017003},"severity":"INFO","message":"UserAgent = gcsfuse/2.3.2-gke.0 (Go version go1.22.4) (GPN:gcsfuse-gke-gcs-fuse-csi) (Cfg:0:0)\n"}
{"timestamp":{"seconds":1724386013,"nanos":891935938},"severity":"INFO","message":"Creating a mount at \"/dev/fd/3\"\n"}
{"timestamp":{"seconds":1724386013,"nanos":891966584},"severity":"INFO","message":"Creating a temporary directory at \"/gcsfuse-buffer/.volumes/data-volume/temp-dir\"\n"}
{"timestamp":{"seconds":1724386013,"nanos":892260906},"severity":"INFO","message":"Creating a new server...\n"}
{"timestamp":{"seconds":1724386013,"nanos":892285943},"severity":"INFO","message":"Set up root directory for bucket planetgpt"}
{"timestamp":{"seconds":1724386014,"nanos":55943033},"severity":"INFO","message":"Mounting file system \"planetgpt\"..."}
{"timestamp":{"seconds":1724386014,"nanos":56292455},"severity":"INFO","message":"File system has been successfully mounted."}
{"timestamp":{"seconds":1724386614,"nanos":56406251},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724386614,"nanos":75877035},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 19.408277ms."}
{"timestamp":{"seconds":1724387214,"nanos":56620258},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724387214,"nanos":78362228},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 21.675598ms."}
{"timestamp":{"seconds":1724387724,"nanos":628448539},"severity":"INFO","message":"Retrying for the error: googleapi: Error 503: We encountered an internal error. Please try again., backendError"}
{"timestamp":{"seconds":1724393814,"nanos":57158309},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724393814,"nanos":73853382},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 16.579936ms."}
{"timestamp":{"seconds":1724394414,"nanos":56095807},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724394414,"nanos":91421975},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 35.273225ms."}
{"timestamp":{"seconds":1724395014,"nanos":56214243},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724395014,"nanos":75738880},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 19.4625ms."}
{"timestamp":{"seconds":1724395614,"nanos":56872616},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724395614,"nanos":76901075},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 19.96629ms."}
{"timestamp":{"seconds":1724396214,"nanos":56622208},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724396214,"nanos":69272882},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 12.579993ms."}
{"timestamp":{"seconds":1724396814,"nanos":56310514},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724396814,"nanos":71821070},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 15.395245ms."}
{"timestamp":{"seconds":1724397113,"nanos":993496341},"severity":"INFO","message":"Retrying for the error: googleapi: Error 503: We encountered an internal error. Please try again., backendError"}
{"timestamp":{"seconds":1724397414,"nanos":56131797},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724397414,"nanos":78600791},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 22.414206ms."}
{"timestamp":{"seconds":1724398014,"nanos":56774250},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724398014,"nanos":72126916},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 15.299395ms."}
{"timestamp":{"seconds":1724398614,"nanos":56168478},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724398614,"nanos":79591122},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 23.365301ms."}
{"timestamp":{"seconds":1724399214,"nanos":56930658},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724399214,"nanos":72334172},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 15.342155ms."}
{"timestamp":{"seconds":1724399814,"nanos":56494677},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724399814,"nanos":71267149},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 14.718696ms."}
{"timestamp":{"seconds":1724400283,"nanos":410530624},"severity":"INFO","message":"Retrying for the error: googleapi: Error 503: We encountered an internal error. Please try again., backendError"}
{"timestamp":{"seconds":1724400414,"nanos":57027786},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724400414,"nanos":78662574},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 21.564104ms."}
{"timestamp":{"seconds":1724401014,"nanos":56111571},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724401014,"nanos":72189338},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 15.964002ms."}
{"timestamp":{"seconds":1724401614,"nanos":56087651},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724401614,"nanos":69288085},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 13.149346ms."}
{"timestamp":{"seconds":1724402214,"nanos":56336451},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724402214,"nanos":76932098},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 20.530066ms."}
{"timestamp":{"seconds":1724402604,"nanos":765366504},"severity":"INFO","message":"Retrying for the error: googleapi: Error 503: Socket write error., backendError"}
{"timestamp":{"seconds":1724402814,"nanos":56972000},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724402814,"nanos":76591045},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 19.559078ms."}
{"timestamp":{"seconds":1724403414,"nanos":56358983},"severity":"INFO","message":"Starting a garbage collection run."}
{"timestamp":{"seconds":1724403414,"nanos":72759865},"severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 16.346854ms."}
when I tries to ls in the mounted directory i got:
ls: reading directory '.': Input/output error
- Attach the screenshot or the logs file to the bug report here. (pasted above)
- If you’re using gcsfuse with any other library/tool/process please list out the steps you took to reproduce the issue.
Additional context Add any other context about the problem here. We were running machine learning model training using pytorch code over zarr files in GCS bucket and that might cause the gcsfuse to fail over time. An enhanced support cases was already created to cloud storage team but there hasnt been any help in last 5 hours. case number: 53082046
SLO: We strive to respond to all bug reports within 24 business hours provided the information mentioned above is included.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22
thank you so so much!
looks like you are right, see output below:
% gcloud storage ls -r gs://bucket-name-hidden/subdir-name-hidden | grep ‘gs://.*//’ gs://bucket-name-hidden/subdir-name-hidden//: gs://bucket-name-hidden/subdir-name-hidden//json/: gs://bucket-name-hidden/subdir-name-hidden//json/ and I will share more details in the support case itself
will try to reproduce that, but that means the current pod will be deleted and replaced by a new pod. Will share results soon.