datafusion: Break datafusion crate into smaller crates
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
a new feature: to break datafusion crate into separate smaller crates.
It helps with code management and dependency reasoning
Describe the solution you’d like
- #1844
- #1772
- file format
- listing and object store
- #1755
- move logical plan builder
- move logical plan structs
- move move optimizers
- #1754
- move physical planner
- move execution exprs
- move physical plan optimizer
- split datafusion-execution sub-module
- move memory and disk manager
- move stats collection
Describe alternatives you’ve considered A clear and concise description of any alternative solutions or features you’ve considered.
Additional context Add any other context or screenshots about the feature request here.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 5
- Comments: 15 (15 by maintainers)
my current plan is to finish items for:
datafusion-commondatafusion-exprbefore release 7:
and finish the rest after the release, to cap the amount of changes in a release
Hi @Jimexist, is it possible to make an exclusive crate for the data source so that it will be easy for the integration of the different kinds of remote object store?
Suppose I hope to introduce the hdfs as one of the remote object store. To be less intrusive, it’s better to make an independent datafusion-objectstore-hdfs crate which depends on the datasource crate of datafusion. Then it would be much easier for other crates to decide whether to depend on the datafusion-hdfs crate or not without cyclic dependencies.