datafusion: Option to disable `Physical input schema should be the same as the one converted from logical schema` error

Is your feature request related to a problem or challenge?

This bug, released in DataFusion 42.0.0 ,

Added a new check in the DefaultPhysicalPlanner that the schema of the output plan is the same as the input plan

https://github.com/apache/datafusion/blob/818ce3f01efe1213a9a1eda5dff1542bb9d457f7/datafusion/core/src/physical_planner.rs#L660-L662

While @jayzhan211 's heroic efforts has this passing in all the DataFusion tests, it turned out this check failed on many downstream implementations:

Downstream in InfluxDB 3.0 we turned the check into a warning in our fork to unblock our upgrade

We even made a patch release to try and get the delta-rs upgrade working:

But it is still failing when I write this (see https://github.com/delta-io/delta-rs/pull/2886#issuecomment-2425616646)

Internal error: Failed due to a difference in schemas, original schema: DFSchema { inner: Schema { fields: [Field { name: “id”, data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “price”, data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “sold”, data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “price_float”, data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “items_in_bucket”, data_type: List(Field { name: “element”, data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “deleted”, data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “__delta_rs_update_predicate”, data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, field_qualifiers: [None, Some(Bare { table: “target” }), None, None, None, None, None], functional_dependencies: FunctionalDependencies { deps: [] } }, new schema: DFSchema { inner: Schema { fields: [Field { name: “id”, data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “price”, data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “sold”, data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “price_float”, data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “items_in_bucket”, data_type: List(Field { name: “item”, data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “deleted”, data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “__delta_rs_update_predicate”, data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, field_qualifiers: [None, Some(Bare { table: “target” }), None, None, None, None, None], functional_dependencies: FunctionalDependencies { deps: [] } }.

Describe the solution you’d like

Note there is at least one open outstanding bug: https://github.com/apache/datafusion/issues/13010

I would like some way to disable this check to unblock upgrades in downstream crates.

Describe alternatives you’ve considered

I propose we add a new config value that lets downstream crates opt in / out of this check, similarly to datafusion.optimizer.skip_failed_rules (see Config Docs)

Something like:

  • datafusion.execution.validate_schema: If true, the DefaultPhysicalPlanner will error if the input plan’s schema does not exactly match the output plan.

Additional context

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 5
  • Comments: 23 (23 by maintainers)

Most upvoted comments

The question I’d want answered is: does disabling this check hide an actual problem or are we confident the problem exists within the check itself?

I believe disabling the check hides actual problems – a list of such problems is here https://github.com/apache/datafusion/issues/12733

However, those plans ran in DataFusion 41 (and sometimes metadata was dropped from the output schema) and now the plans refuse to run.

So depending on your perspective this is either an improvement or regression:

  • It is a regression for a user running SQL queries perspective because a query that used to run great no longer does
  • It is an improvement for a developer as it now makes it easy to find queries that violate invariants

So having a flag to change behaviors based on your point of view I think makes sense

That is, do we know for certain that disabling the check isn’t allowing faulty output? In general I recommend against features to disable checks but I also recognize that unblocking downstream users may be more important.

No, we do not know this for certain. I think my point of view is that running with the check disabled is no worse than DataFusion 41 so while not ideal, it isn’t worse

@wiedld and think we have at least one more issue that is as yet unfiled (that we see occuring in our logs after we upgraded DataFusion and turned this error into a warning). More to come – we are working on getting a self contained reproducer

that would be tremendously helpful! and as @wiedld pointed out, would ensure the bug is indeed in DF code.

Thus far we have found 1 additional schema mishandling bug, triggering this check, where the bug was in datafusion. If I encounter more bugs in DF code, I’ll add it to the Epic https://github.com/apache/datafusion/issues/12733.

My vision we should correctly implement Eq for Schema or provide a method that checks everything but nullability and metadata.

I think it is somewhat debatable if we should/shouldn’t be checking that the nullability and metadata matches. I actually think having the invariant that the corresponding physical plan created for a LogicalPlan should have the exact same schema seems quite reasonable 🤔

All currently known looks like solved, right? Or am i looking at this the wrong way?

I think it is more accurate to say “all currently filed issues have been resolved”

know / filed – it seems like we mean mostly the same 👍

we are working on getting a self contained reproducer

thanks! that would be tremendously helpful! and as @wiedld pointed out, would ensure the bug is indeed in DF code.

@wiedld are you able to capture failing queries (or rather: queries giving a warning) and maybe turn some into actionable issues for the community?

@findepi I’ll try to get us actionable issues this week. Specifically, I need to isolate reproducers in either our code base (e.g. our physical optimizers) versus datafusion.

It is probably happens when users utilize only physical planner, although there are schema alignments that happening on logical planner?

There are two classes of errors we have seen so far in InfluxDB:

  1. Null declarations as predicated by @comphead
  2. Metadata mismatches (metadata is dropped by some physical optimizer)

I believe delta-rs is also seeing field name mismatches (e.g. the embedded fieldname on ListArray is "item" vs "element" or something)

I remember bunch of issues on schema comparison when different null flag for the column caused a problem.Since DF reworked the Schema methods this issue might come up again

We (InfluxData) will likely contribute fixes back upstream into DataFusion as we find issues as well