OpenSearch: [BUG] Breaking changes with opensearch.http.override_main_response_version=true

That would be the upgrade path: upgrade client to 2.0, it should work for both versions, than start upgrading clusters.

That is the correct upgrade path for clients depending on the HLRC.

opensearch.http.override_main_response_version=true doesn’t impact the node connectivity with ES 7.10

Correct. opensearch.http.override_main_response_version=true does not impact internal node connectivity it was only added to ensure compatibility w/ legacy 7.x clients. I’m sorry if that wasn’t clear. My point is that 2.0.0 supports Transport API compatibility back to 7.10.0. The HLRC (client facing API) does not include Version.onOrAfter logic so it doesn’t subscribe to the same compatibility guarantees; it follows the same deprecation / removal path as features which is why the types endpoints were removed (after being deprecated in 7.x/1.x).

I’m not sure the project should be shy about 2.0.0 having breaking changes

💯

there is some dissonance between OpenSearch 2.0.0 saying it’s 7.10.2 yet not having the same API

Agree. I’ve slept since we removed the transport client December of last year so we don’t need to retain transport client compatibility (nor should the REST client spoofing matter here).

+1 for removing the override_main_response_version property and forcing client upgrades.

Can anyone pick this up?

opensearch-project/OpenSearch#3031

Many users of Beats and Logstash have switched to OpenSearch and Elastic has made it clear that these products do not intend to support OpenSearch. There are explicit version checks to verify compatibility with Elasticsearch that were circumvented using the spoofing mechanism. Without the spoof users of some filebeat processors are now DOA after upgrading to 2.0.

I have opened a PR to revert the removal of spoofing so these affected users can continue to function by explicitly setting the version. Yes, breaking compatibility with upgraded clients might be a side effect if users explicitly set this setting to function with older versions of Beats and Logstash. For this reason we need the REST API Versioning mechanism so this will, hopefully, be a temporary situation. Longer term we need a plan to either fork beats and logstash or replace it with an alternative like FluentBit.

That is the correct upgrade path for clients depending on the HLRC.

but is the upgrade path from ES 7.10.2 HLRC and/or OpenSearch 1.x HLRC to OpenSearch 2.x HLRC or to opensearch-java? elastic said their elasticsearch-java is now the official java client and should be used instead and e.g. spring-data-elasticsearch is now refactoring to support both clients (and eventually also an OpenSearch client): https://github.com/spring-projects/spring-data-elasticsearch/issues/1973 & https://github.com/spring-projects/spring-data-elasticsearch/issues/1853

upgrading from one HLRC to another HLRC is probably easiest (for those projects using it directly) but moving directly to opensearch-java is probably the more long-term approach. and since opensearch-java is not part of OpenSearch itself i would presume it would lend itself much more to being compatible over multiple OpenSearch major versions than the HLRC which is an integral part of OpenSearch and makes use of internal OpenSearch libraries?

@saratvemulapalli I think it is reversed: Rest clients which support 2.0 and we do want to support them for 7.10. That would be the upgrade path: upgrade client to 2.0, it should work for both versions, than start upgrading clusters.

We use a couple supporting tools that still depend on the 7.x ES client libraries. We look forward to disabling the 7.10.2 spoofing as soon as possible to remove our reliance on this workaround.

I based our plans on the answer I received at the community meeting on 2021-10-19: “Do we expect the ES version compatibility flag to be removed in 2.0?” Kyle’s answer was to expect the version compatibility flag to be re-evaluated prior to 3.0. If removed, it would be removed no earlier than 3.0.

If this is no longer possible and it becomes necessary for us to remain on 1.x until we no longer rely on the workaround, will 1.x continue to get security patches for while?