beam: [Bug]: ReadAllFiles does not fully read gzipped files from GCS
What happened?
Since the refactor of gcsio (2.52?) ReadAllFiles does not fully read gzipped files from GCS. Part of the file will be correctly returned but rest will go missing.
I presume this is caused by the fact that GCS performs decompressive transcoding while _ExpandIntoRanges uses the GCS objects metadata to determine the read range. This means that the file size we receive is larger than the maximum of the read range.
For example, a gzip on GCS might have a file size of 1 MB and this will be the object size in the metadata. Thus the maximum of the read range will be 1 MB. However, when beam opens the file it’s already decompressed by GCS so the file size will be 1.5 MB and we won’t read 0.5 MB out of it thus causing data loss.
Issue Priority
Priority: 1 (data loss / total loss of function)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 34 (19 by maintainers)
I see. We need to check decompressive transcoding for the GCS file to determine whether the content is compressed rather than relying on the file extension.
This only loads 75,601 lines.
#19413 could be related for uploading the file to GCS.
Great! GCS decompressive transcoding is a bit unintuitive to Beam users here, and when it happens, we see data loss. I think it is more natural to expect users to specify GZIP or AUTO in those cases rather than UNCOMPRESSED, as shown in the proposal.
Yep, that’s the idea, but it is implemented differently in the proposed fix (#33384).
accept-encoding:gzipis always set in the request header: https://github.com/googleapis/python-storage/blob/4e9a382dd91f7d951cb9b95319b05e0cdda06415/google/cloud/storage/blob.py#L4286). It then adds an extra decoding step in itself (https://github.com/googleapis/google-resumable-media-python/blob/402feb7b38a972daad9bd3e26b80ddd0879bd53f/google/resumable_media/requests/download.py#L127) before returning the file content to the caller (in this case Beam). It merely mimicks the effects of server-side decompressive transcoding; the actual decoding workload is done on the client and we are NOT leveraging the real GCS server-side transcoding at all.In my fix (#33384), I let GCS client library skip the decoding step in itself and rely on Beam’s decoding mechanism in CompressedFile to process the file. I believe this is more intuitive to our users and the end result of this approach is exactly the same as what has been proposed in the previous table.