delta-rs: Failed to apply transaction log: Invalid JSON in log record while reading delta table
databricks runtime version 10.1
version: deltalake version = 0.5.3
Binding:
Environment:
- AWS:
- databricks runtime:
- Other:
Bug
While reading the deltatable using deltalake python library, getting Failed to apply transaction log: Invalid JSON in log record error.
What you expected to happen: It should read and be able to list the files. Spark.read.format(‘delta’) is able to read it fine, so doesn’t look like a bug in the file
How to reproduce it: Couldn’t reproduce this in my environment.
More details: Steps tried to debug the issue
- Tried reading it using
spark.read.format('delta'), its working fine - Tried reading the json file inside _delta_log folder using
spark.read.json(), can read it fine - Tried to read other delta table using this package, can read it and list all the files
- Tried to move the
00000000000000000000.jsonfile from _delta_log folder to another folder, and read it using deltatable package, but got the same error
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 26 (11 by maintainers)
Great news! I finally was able to test. I am able to list files now. Thank you!