delta-rs: Failed to apply transaction log: Invalid JSON in log record while reading delta table

databricks runtime version 10.1

version: deltalake version = 0.5.3

Binding:

Environment:

  • AWS:
  • databricks runtime:
  • Other:

Bug

While reading the deltatable using deltalake python library, getting Failed to apply transaction log: Invalid JSON in log record error.

What you expected to happen: It should read and be able to list the files. Spark.read.format(‘delta’) is able to read it fine, so doesn’t look like a bug in the file

How to reproduce it: Couldn’t reproduce this in my environment.

More details: Steps tried to debug the issue

  • Tried reading it using spark.read.format('delta'), its working fine
  • Tried reading the json file inside _delta_log folder using spark.read.json(), can read it fine
  • Tried to read other delta table using this package, can read it and list all the files
  • Tried to move the 00000000000000000000.json file from _delta_log folder to another folder, and read it using deltatable package, but got the same error

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 26 (11 by maintainers)

Commits related to this issue

Most upvoted comments

Great news! I finally was able to test. I am able to list files now. Thank you!