OpenSearch: [opensearch-plugin] Cannot install old patch version of plugins on newer opensearch builds
2021-12-10 21:54:54 INFO Executing "/tmp/tmp3_3h_wbz/opensearch-1.2.1/bin/opensearch-plugin install --batch file:/tmp/tmp3_3h_wbz/opensearch-job-scheduler-1.2.0.0.zip" in /tmp/tmp3_3h_wbz/opensearch-1.2.1
-> Installing file:/tmp/tmp3_3h_wbz/opensearch-job-scheduler-1.2.0.0.zip
-> Downloading file:/tmp/tmp3_3h_wbz/opensearch-job-scheduler-1.2.0.0.zip
-> Failed installing file:/tmp/tmp3_3h_wbz/opensearch-job-scheduler-1.2.0.0.zip
-> Rolling back file:/tmp/tmp3_3h_wbz/opensearch-job-scheduler-1.2.0.0.zip
-> Rolled back file:/tmp/tmp3_3h_wbz/opensearch-job-scheduler-1.2.0.0.zip
Exception in thread "main" java.lang.IllegalArgumentException: Plugin [opensearch-job-scheduler] was built for OpenSearch version 1.2.0 but version 1.2.1 is running
at org.opensearch.plugins.PluginsService.verifyCompatibility(PluginsService.java:390)
at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:803)
at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:858)
at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:263)
at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:237)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)
This is blocking us from consuming the redistribution artifacts for patch releases.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 34 (21 by maintainers)
Commits related to this issue
- Add support for dependencies in plugin descriptor properties with semver range (#1707) Signed-off-by: Abhilasha Seth <abseth@amazon.com> — committed to abseth-amzn/OpenSearch by abseth-amzn 7 months ago
Thanks for a thoughtful proposal @abseth-amzn! Tl;dr I would prefer (3) + (1) and I think it’s a similar amount of work as just (1).
Longer story:
My preference would be for the longest term solution, aka (3) where plugins specify, for example,
~> 2.xby default (compatible with any 2.x version) and lock that down only if they find an incompatibility for whatever reason (e.g.>= 2.0, <= 2.3).Because OpenSearch does follow semver, I think you could default to (1) if a plugin does not specify its compatibility, and introduce (3) later. It’s weird that the dependency (server) would specify version compatibility for its dependent (plugin), but I think it’s OK as a default when the dependent has no opinion.
Finally, I would not assume that (1) is less work than (3). I think that 99% of the work is on the server anyway, the only difference is that the plugin needs an additional configuration field for which version(s) of the server it’s compatible with.
The more I thought about this move the less I like the idea. Testing would be horrible in this situation, and I cant find a good way around it.
I agree with the following statement the most as it is also this exact case that has brought this issue to light:
+1 @abseth-amzn I like the proposal, it makes sense to relax patch versions for plugins and longer term extensions will support running across versions including major and minor.
We have seen examples in the past where patch versions have caused problems:
As most of the comments here already put an outline, we’d like to see:
>2.5.1, <2.5.7We’d like to start making changes to make this happen. Mostly everybody looks aligned with the discussion, let us know if there is any additional feedback while we start moving the needle for patch versions. cc: @dblock @reta @Bukhtawar @nknize @shwetathareja @rursprung @krishna-ggk @peternied
I’ve opened up a meta issue which will solve this for the longer term: #2283 and enable hot-swap of plugins/extensions.
hot-swap of plugins is on the radar of @saratvemulapalli
As a first step towards supporting semVer range of compatible versions for plugins, we could start with relaxing the patch version check (for default compatibility range).
This would require building the ability to test across patch versions of the core for every patch release of a plugin and vice versa. This would include testing a plugin with compiled version of newer patch of OpenSearch (and vice versa) along with running integration tests to catch runtime issues.
with this you mean a semver definition like
> 2.0.0, < 2.8, right? then this would be great news as it’d solve 99% of the problems encountered as plugins can then start using that once they upgrade to the opensearch version which supports this and then don’t have to deal with it again until the next major releaseConsider the case where we have version 1.3.3 out there and mistakenly commit a (breaking) change in 1.3.4. Plugin update relies on that feature, which then gets reverted in 1.3.5. Plugin may choose to first lock down its max and quickly release, before addressing the problem at the root.
What I am saying is that developers need control over compatibility from the plugin POV, leave it to them to figure out how they use it. We should introduce restrictions only where it’s obviously harmful.
Good point! The way forward I see with this is a clear documentation on
opensearch.api/opensearch.internalon Java docs and strong BWC checks to ensure these contracts are honoured at patch and minor version bump ups. Any plugin user before they can specify semVer ranges for compatibility need to ensure via some gradle checks that they don’t strictly depend on anything that is notopensearch.apiand show them warnings on usages if any such dependency is found.Having these checks on the core and the plugin package can definitely help avoid surprises.
Yes, there shouldn’t be any breaking changes in plugin interfaces in minor version releases. But there are lot of internal classes/ objects exposed through these interfaces e.g. DiscoveryPlugin which has bunch of methods and expose other internal classes like TransportService/ NetworkService etc. and similarly ClusterService is exposed in a different ClusterPlugin interface. These internal classes are not bound by any plugin interface contract. And, if a plugin depends on their methods (which in turn can return other internal classes) and these are changed, then it can cause potential incompatibility or failures. This can be obvious if there is a compile time failure or just a behavior change, would figure out during runtime testing. So providing configuration to users to specify the plugin version range compatibility is one but real responsibility lies with testing. I found similar discussion for core extension points - https://github.com/opensearch-project/OpenSearch/issues/2868
Hello all, I am working on supporting custom plugins with Amazon’s managed OpenSearch service. It would be desirable to allow customers to upgrade their domain’s engine version without requiring them to provide updated custom plugins every time, especially for patch version upgrades for the core. I have read through the above mentioned comments and see following options for the path forward:
I wanted to explore the possibility of starting with #1 to allow for seamless version upgrade experience for customers using custom plugins. I understand that it would require additional testing effort but asking customers to provide custom plugins with updated versions (when most of the time there is not much plugin code change) may create significant friction in feature adoption. Looking for thoughts from the community on this.
IMHO in your example the two plugins shouldn’t restrict to a specific minor version. if the plugins use proper semver they can just rely on e.g.
>= 1.2.2which then accepts anything<2.0.0.but i think your example with plugins depending on each other shows something else: it should be possible to do an in-place upgrade of a plugin rather than having to uninstall the old version and then install the new one again. then you don’t need to uninstall dependent plugins first (unless they’re incompatible with the new version). i guess that’s another feature request to spin out of this discussion?
IMHO managing such dependencies is a solved problem - all package managers (on os-level like apt, software development dependency managers like npm, etc.) deal exactly with this. esp. the os-level ones (e.g. apt) need to deal with it in a similar way opensearch is doing it: they need to be able to replace existing packages with new ones while others are depending on them and ensure that upgrades are done in the right order (and all of which while the system is running, which is more complicated than in the opensearch use-case where the opensearch node is not running during the installation).
I agree with the conclusion being reached here. OpenSearch plugins are semi-independent of the rest of OpenSearch, but there aren’t currently good mechanisms to prevent test-matrix explosion aside strict version-locking. Relaxing the version-locking requirement in a meaningful way is more than a change to the build process.
IMO this would be a great problem to tackle: how could we really decouple plugin deployment in a way that is user-friendly and developer-friendly.