datafusion: [DISCUSS] DataFusion less frequent major / breaking releases (ease using multiple third-party extensions (like delta, or iceberg) )
Is your feature request related to a problem or challenge?
One of the dreams of the composable data ecosystem is to quickly assemble a system from various components (DataFusion, data formats
DataFusion still releases once a month, which allows code to quickly flow but also causes at least 2 challenges:
- Takes non trivial work required to upgrade downstream projects, as mentioned in https://github.com/apache/datafusion/issues/5269
- Make upgrading and using downstream third-party extensions hard
Third party extensions like delta-rs and iceberg provide TableProviders for DataFusion, which is really nice. However, to use those packages the versions of DataFusion must match exactly.
This means for an application that relies on multiple downstream packages must wait until ALL of them have upgraded to the new version in order to upgrade DataFusion. If there is any delay in the downstream libraries updating, it delays.
For example, an application that wants to use delta-rs, iceberg, and the table-providers crate, there is a race after each upgrade of DataFusion
Let’s take a release timeline for
- +0 days: DataFusion version
Xreleased - +7 days: New delta-rs releases upgraded to DataFusion
X - +11 days: new iceberg crate released upgraded to DataFusion
X - +12 days: new table-providers version is released
- +13-30 days: End user app can upgrade DataFusion and delta, and icerberg
- +31 days: New DataFusion is released again
Describe the solution you’d like
I would like downstream libraries to have more time and schedule flexibility when upgrading DataFusion and other dependent crates, so that it is easier to construct a system from different components
Describe alternatives you’ve considered
Option 1: Switch to major/minor release cadence
We could follow the model of arrow-rs which does releases monthly, but breaking releases only quarterly. Here is how it works in arrow-rs: https://github.com/apache/arrow-rs?tab=readme-ov-file#release-versioning-and-schedule
This would mean continuing to release every month, but only allowing breaking API changes every 3rd release (or some other cadence)
The major cost here is that maintainers and contributors would have to be diligent about not merging breaking API changes until a major release
This is possible to automate somewhat:
- https://github.com/apache/datafusion/pull/16078 from @logan-keede
- https://github.com/apache/datafusion/pull/16541 from @lic
Option 2: LTS and feature branch
-Keep (at least) two branches going: LTS and main, as proposed by @andygrove in https://github.com/apache/datafusion/issues/5269
In this model we would likely backport changes to the LTS branch and make releases from there. The downside of this approach is that there is extra work to backport changes to LTS.
Additional context
No response
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 15 (14 by maintainers)
The responsibility for creating the PRs to backport fixes to the release branch should fall to the downstream users who are waiting on those fixes. There is additional work for DataFusion maintainers in terms of PR reviews though.
Related is I think we will do a 48.0.1 release:
That basically follows the model of the LTS idea: we’ll backport relevant fixes only.
This is basically the same cadence that we use in arrow-rs as well. The only potential issue I see with doing something like that for DataFusion is accumulating 3 months of breaking changes might be a lot 🤔
if I may, duckdb process seems a bit reasonable, release a major version with breaking change every three months (1.1, 1.2 , 1.3 etc), release a bug fix every month, 1.3.1 , 1.3.2 then next version will be 1.4, and they have nightly build for both the main branch and the stable branch
And to be clear, in my mind option 1 still has us doing monthly releases, we just restrict which releases have breaking API changes
IMHO, option 1 may be better, it is not as frequent as current release cycle but still frequent enough to take breaking changes in relatively small and digestible chunks.
Not sure if option 2 would force downstream project to have same branching strategy. Otherwise it would bring “big update” every time LTS was released.