tensorflow: tf.io.gfile.glob does not list all files in a Google Cloud Storage bucket

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): ?
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: /
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): 2.0.0
  • Python version: 3
  • Bazel version (if compiling from source): /
  • GCC/Compiler version (if compiling from source): /
  • CUDA/cuDNN version: /
  • GPU model and memory: /

Describe the current behavior When listing file with tf.io.gfile.glob not all images are returned. It seems it is not resolving the folders recursively. When using the same path with gsutils we get the correct image count.

Describe the expected behavior When using the same gs:// path with gsutils we get the correct amount of images.

Code to reproduce the issue In order to reproduce the behavior I prepared a Google Bucket with the following structure. The bucket is public accessible, please feel free to use it to reproduce the behavior on your end: gs://tensorflow-issue-reproduction

level0 level1 level2 level3

In summary we have 4 jpg images nested in different folder levels.

TensorFlow 2 code to reproduce

files = tf.io.gfile.glob('gs://tensorflow-issue-reproduction/**/*.jpg')
print('file count: ', len(files))
# found files 1

gsutil command which works properly

gsutil du gs://tensorflow-issue-reproduction/**/*.jpg | wc -l
# found files 4

Other info / logs /

Best regards Sascha

About this issue

  • Original URL
  • State: open
  • Created 7 years ago
  • Comments: 16 (3 by maintainers)

Most upvoted comments

Unfortunately we didn’t yet get to this issue. Apologies for the delay

Hi,

Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.

The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.

Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.