datafusion: Option to disable `Physical input schema should be the same as the one converted from logical schema` error
Is your feature request related to a problem or challenge?
This bug, released in DataFusion 42.0.0 ,
Added a new check in the DefaultPhysicalPlanner that the schema of the output plan is the same as the input plan
While @jayzhan211 's heroic efforts has this passing in all the DataFusion tests, it turned out this check failed on many downstream implementations:
- https://github.com/apache/datafusion/issues/12733 lists issues we hit during our upgrade in InfluxDB 3.0
- https://github.com/delta-io/delta-rs/pull/2886 @ion-elgreco and @rtyler and @Xuanwo while updating delta.rs to DataFusion 42.0.0
Downstream in InfluxDB 3.0 we turned the check into a warning in our fork to unblock our upgrade
We even made a patch release to try and get the delta-rs upgrade working:
But it is still failing when I write this (see https://github.com/delta-io/delta-rs/pull/2886#issuecomment-2425616646)
Internal error: Failed due to a difference in schemas, original schema: DFSchema { inner: Schema { fields: [Field { name: “id”, data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “price”, data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “sold”, data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “price_float”, data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “items_in_bucket”, data_type: List(Field { name: “element”, data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “deleted”, data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “__delta_rs_update_predicate”, data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, field_qualifiers: [None, Some(Bare { table: “target” }), None, None, None, None, None], functional_dependencies: FunctionalDependencies { deps: [] } }, new schema: DFSchema { inner: Schema { fields: [Field { name: “id”, data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “price”, data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “sold”, data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “price_float”, data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “items_in_bucket”, data_type: List(Field { name: “item”, data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “deleted”, data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: “__delta_rs_update_predicate”, data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, field_qualifiers: [None, Some(Bare { table: “target” }), None, None, None, None, None], functional_dependencies: FunctionalDependencies { deps: [] } }.
Describe the solution you’d like
Note there is at least one open outstanding bug: https://github.com/apache/datafusion/issues/13010
I would like some way to disable this check to unblock upgrades in downstream crates.
Describe alternatives you’ve considered
I propose we add a new config value that lets downstream crates opt in / out of this check, similarly to datafusion.optimizer.skip_failed_rules (see Config Docs)
Something like:
datafusion.execution.validate_schema: If true, theDefaultPhysicalPlannerwill error if the input plan’s schema does not exactly match the output plan.
Additional context
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 5
- Comments: 23 (23 by maintainers)
I believe disabling the check hides actual problems – a list of such problems is here https://github.com/apache/datafusion/issues/12733
However, those plans ran in DataFusion 41 (and sometimes metadata was dropped from the output schema) and now the plans refuse to run.
So depending on your perspective this is either an improvement or regression:
So having a flag to change behaviors based on your point of view I think makes sense
No, we do not know this for certain. I think my point of view is that running with the check disabled is no worse than DataFusion 41 so while not ideal, it isn’t worse
Thus far we have found 1 additional schema mishandling bug, triggering this check, where the bug was in datafusion. If I encounter more bugs in DF code, I’ll add it to the Epic https://github.com/apache/datafusion/issues/12733.
I think it is somewhat debatable if we should/shouldn’t be checking that the nullability and metadata matches. I actually think having the invariant that the corresponding physical plan created for a LogicalPlan should have the exact same schema seems quite reasonable 🤔
know / filed – it seems like we mean mostly the same 👍
thanks! that would be tremendously helpful! and as @wiedld pointed out, would ensure the bug is indeed in DF code.
@findepi I’ll try to get us actionable issues this week. Specifically, I need to isolate reproducers in either our code base (e.g. our physical optimizers) versus datafusion.
There are two classes of errors we have seen so far in InfluxDB:
I believe delta-rs is also seeing field name mismatches (e.g. the embedded fieldname on
ListArrayis"item"vs"element"or something)I remember bunch of issues on schema comparison when different null flag for the column caused a problem.Since DF reworked the
Schemamethods this issue might come up againWe (InfluxData) will likely contribute fixes back upstream into DataFusion as we find issues as well