arrow: [C++][Parquet] Timestamp conversion from Parquet to Arrow does not follow compatibility guidelines for convertedType
Describe the bug, including details regarding any error messages, version, and platform.
The Parquet format document indicates that converted types should have isAdjustedToUtc=true for backward compatibility, but the current C++ implementation sets this to false for converted types.
The Rust implementation already follows the spec as described and the Go implementation has #39467 open to correct this.
Component(s)
C++, Parquet
About this issue
- Original URL
- State: closed
- Created 6 months ago
- Comments: 20 (20 by maintainers)
Commits related to this issue
- GH-39489: change ConvertType Thrift Timestamp to adjustUtc — committed to mapleFU/arrow by mapleFU 6 months ago
- GH-39489: [C++][Parquet] ConvertedType TIME and TIMESTAMP should imply adjustedToUtc (#39491) ### Rationale for this change TLDR: When reading from a Parquet file with legacy timestamp not written... — committed to apache/arrow by mapleFU 6 months ago
- GH-39489: [C++][Parquet] Revert #39491 and add timestamp behavior to doc — committed to mapleFU/arrow by mapleFU 6 months ago
- GH-39489: [C++][Parquet] Revert #39491 and add timestamp behavior to doc — committed to mapleFU/arrow by mapleFU 6 months ago
- GH-39489: [C++][Parquet] Revert #39491 and add timestamp behavior to doc — committed to mapleFU/arrow by mapleFU 6 months ago
- GH-39466: [Go][Parquet] Align Arrow and Parquet Timestamp Instant/Local Semantics (#39467) ### Rationale for this change Closes: #39466 ### What changes are included in this PR? - Update logic ... — committed to apache/arrow by joellubi 5 months ago
- GH-39489: [C++][Parquet] ConvertedType TIME and TIMESTAMP should imply adjustedToUtc (#39491) ### Rationale for this change TLDR: When reading from a Parquet file with legacy timestamp not written... — committed to clayburn/arrow by mapleFU 6 months ago
- GH-39466: [Go][Parquet] Align Arrow and Parquet Timestamp Instant/Local Semantics (#39467) ### Rationale for this change Closes: #39466 ### What changes are included in this PR? - Update logic ... — committed to clayburn/arrow by joellubi 5 months ago
- GH-39489: [C++][Parquet] ConvertedType TIME and TIMESTAMP should imply adjustedToUtc (#39491) ### Rationale for this change TLDR: When reading from a Parquet file with legacy timestamp not written... — committed to dgreiss/arrow by mapleFU 6 months ago
- GH-39466: [Go][Parquet] Align Arrow and Parquet Timestamp Instant/Local Semantics (#39467) ### Rationale for this change Closes: #39466 ### What changes are included in this PR? - Update logic ... — committed to dgreiss/arrow by joellubi 5 months ago
- GH-39489: [C++][Parquet] ConvertedType TIME and TIMESTAMP should imply adjustedToUtc (#39491) ### Rationale for this change TLDR: When reading from a Parquet file with legacy timestamp not written... — committed to zanmato1984/arrow by mapleFU 6 months ago
- GH-39466: [Go][Parquet] Align Arrow and Parquet Timestamp Instant/Local Semantics (#39467) ### Rationale for this change Closes: #39466 ### What changes are included in this PR? - Update logic ... — committed to zanmato1984/arrow by joellubi 5 months ago
- GH-39466: [Go][Parquet] Align Arrow and Parquet Timestamp Instant/Local Semantics (#39467) ### Rationale for this change Closes: #39466 ### What changes are included in this PR? - Update logic ... — committed to thisisnic/arrow by joellubi 5 months ago
Some related discussions:
Now I suppose that because of those mentioned issues, when converting a TimestampLogicalType to an Arrow timestamp type, we do ignore the
isAdjustedToUtcflag if the logical type has been annotated to come from a converted type. So then changingisAdjustedToUtcin those cases shouldn’t actually impact any behaviour in the Arrow API layer of the Parquet library?