tensorflow: tf.io.gfile.glob does not list all files in a Google Cloud Storage bucket
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): ?
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: /
- TensorFlow installed from (source or binary): binary
- TensorFlow version (use command below): 2.0.0
- Python version: 3
- Bazel version (if compiling from source): /
- GCC/Compiler version (if compiling from source): /
- CUDA/cuDNN version: /
- GPU model and memory: /
Describe the current behavior
When listing file with tf.io.gfile.glob not all images are returned. It seems it is not resolving the folders recursively.
When using the same path with gsutils we get the correct image count.
Describe the expected behavior When using the same gs:// path with gsutils we get the correct amount of images.
Code to reproduce the issue
In order to reproduce the behavior I prepared a Google Bucket with the following structure. The bucket is public accessible, please feel free to use it to reproduce the behavior on your end: gs://tensorflow-issue-reproduction

In summary we have 4 jpg images nested in different folder levels.
TensorFlow 2 code to reproduce
files = tf.io.gfile.glob('gs://tensorflow-issue-reproduction/**/*.jpg')
print('file count: ', len(files))
# found files 1
gsutil command which works properly
gsutil du gs://tensorflow-issue-reproduction/**/*.jpg | wc -l
# found files 4
Other info / logs /
Best regards Sascha
About this issue
- Original URL
- State: open
- Created 7 years ago
- Comments: 16 (3 by maintainers)
Unfortunately we didn’t yet get to this issue. Apologies for the delay
Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.