gcs-fuse-csi-driver: Synchronization Issue between gcsfuse and Kubernetes Pod: 'No Such File or Directory' Error on File Update

I am encountering a synchronization issue between gcsfuse and a pod in Kubernetes environment. When I update the files in the Google Cloud Storage (GCS) bucket mounted by gcsfuse, the gcs fuse sidecar fails to access the updated files and throws an error.

Error Logs:

fuse: *fuseops.ReadFileOp error: no such file or directory
ReadFile: no such file or directory, fh.reader.ReadAt: startRead: NewReader: storage: object doesn't exist

Configuration:

GKE Pod: Mounted GCS bucket using gcsfuse with the following configuration: mountOptions: ‘uid=101,gid=82’

Kubernetes Job: Updates the contents of the GCS bucket by replacing the existing files. Names are not changed.

Observations and Troubleshooting Steps Taken:

  • Verified that the files are successfully uploaded to the GCS bucket by the Kubernetes Job.
  • Confirmed that the gcsfuse volume is correctly mounted in the pod.
  • Ensured consistent file naming between the updated files and the files accessed by the GKE Pod.
  • Verified file system permissions and confirmed that the uid and gid specified in the mountOptions match the permissions required by the Pod.
  • Tried setting stat-cache-ttl=0, type-cache-ttl=0, implicit-dirs in the mount options

Anything that I’m missing here?

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (9 by maintainers)

Most upvoted comments

Any solution to this issue ? We have the same situation. In our case access to GCSFuse mounted volume works well if we have only one pod replica. In case of 2 replicas unfortunately it seems that the second replica is not able to immediately see what is updated by the first.