gcs-fuse-csi-driver: Synchronization Issue between gcsfuse and Kubernetes Pod: 'No Such File or Directory' Error on File Update
I am encountering a synchronization issue between gcsfuse and a pod in Kubernetes environment. When I update the files in the Google Cloud Storage (GCS) bucket mounted by gcsfuse, the gcs fuse sidecar fails to access the updated files and throws an error.
Error Logs:
fuse: *fuseops.ReadFileOp error: no such file or directory
ReadFile: no such file or directory, fh.reader.ReadAt: startRead: NewReader: storage: object doesn't exist
Configuration:
GKE Pod: Mounted GCS bucket using gcsfuse with the following configuration: mountOptions: ‘uid=101,gid=82’
Kubernetes Job: Updates the contents of the GCS bucket by replacing the existing files. Names are not changed.
Observations and Troubleshooting Steps Taken:
- Verified that the files are successfully uploaded to the GCS bucket by the Kubernetes Job.
- Confirmed that the gcsfuse volume is correctly mounted in the pod.
- Ensured consistent file naming between the updated files and the files accessed by the GKE Pod.
- Verified file system permissions and confirmed that the uid and gid specified in the mountOptions match the permissions required by the Pod.
- Tried setting stat-cache-ttl=0, type-cache-ttl=0, implicit-dirs in the mount options
Anything that I’m missing here?
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 17 (9 by maintainers)
Any solution to this issue ? We have the same situation. In our case access to GCSFuse mounted volume works well if we have only one pod replica. In case of 2 replicas unfortunately it seems that the second replica is not able to immediately see what is updated by the first.