OpenSearch: [BUG] Trying to upgrade from elasticsearch 7.0. ” Received message from unsupported version”
Describe the bug
Trying to upgrade from elasticsearch 7.0. ” Received message from unsupported version”
Im working on trying to upgrade an existing cluster (es 7.0.0) to OpenSearch, by using a rolling upgrade process, I have a new OpenSearch node built running 1.3.1 (also tried 1.3.2), and when I start the node up, I’m getting and “Received message from unsupported version” message.
I’m sure I’m missing something obvious here.
log output:
[2022-05-05T11:19:48,867][INFO ][o.o.c.c.JoinHelper ] [os-staging-01.duh-stage] failed to join {ip-10-49-1-181.us-west-2.compute.internal}{SbdvE2nbRC-Sa34AAsJrYA}{2A0qVPcGR3KzLNxZeSD9Fw}{10.49.1.181}{10.49.1.181:9300}{dim}{ml.machine_memory=33072087040, rack=us-west-2a, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={os-staging-01.duh-stage}{ebEUk3R2RVmRnxKOtQxEew}{nHIFHs4HR7GpbT5U9eeF_w}{10.49.1.136}{10.49.1.136:9300}{dim}{rack=us-west-2a, shard_indexing_pressure_enabled=true}, minimumTerm=4, optionalJoin=Optional.empty}
org.opensearch.transport.RemoteTransportException: [ip-10-49-1-181.us-west-2.compute.internal][10.49.1.181:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.transport.ConnectTransportException: [os-staging-01.duh-stage][10.49.1.136:9300] general node connection failure
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener$1.onFailure(TcpTransport.java:1284) ~[?:?]
at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:138) ~[?:?]
at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:115) ~[?:?]
at org.elasticsearch.transport.TcpTransport$1.doRun(TcpTransport.java:975) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[?:?]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[?:?]
at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:967) ~[?:?]
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:942) ~[?:?]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[?:?]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.IllegalStateException: Received message from unsupported version: [35.24.79-beta2] minimal compatible version is: [6.7.0]
at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:139) ~[?:?]
at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:115) ~[?:?]
at org.elasticsearch.transport.TcpTransport$1.doRun(TcpTransport.java:975) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[?:?]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[?:?]
at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:967) ~[?:?]
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:942) ~[?:?]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[?:?]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
at java.lang.Thread.run(Thread.java:835) ~[?:?]
[2022-05-05T11:19:48,872][INFO ][o.o.c.c.JoinHelper ] [os-staging-01.duh-stage] failed to join {ip-10-49-1-181.us-west-2.compute.internal}{SbdvE2nbRC-Sa34AAsJrYA}{2A0qVPcGR3KzLNxZeSD9Fw}{10.49.1.181}{10.49.1.181:9300}{dim}{ml.machine_memory=33072087040, rack=us-west-2a, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={os-staging-01.duh-stage}{ebEUk3R2RVmRnxKOtQxEew}{nHIFHs4HR7GpbT5U9eeF_w}{10.49.1.136}{10.49.1.136:9300}{dim}{rack=us-west-2a, shard_indexing_pressure_enabled=true}, minimumTerm=4, optionalJoin=Optional.empty}
org.opensearch.transport.RemoteTransportException: [ip-10-49-1-181.us-west-2.compute.internal][10.49.1.181:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.transport.ConnectTransportException: [os-staging-01.duh-stage][10.49.1.136:9300] general node connection failure
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener$1.onFailure(TcpTransport.java:1284) ~[?:?]
at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:138) ~[?:?]
at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:115) ~[?:?]
at org.elasticsearch.transport.TcpTransport$1.doRun(TcpTransport.java:975) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[?:?]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[?:?]
at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:967) ~[?:?]
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:942) ~[?:?]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[?:?]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.IllegalStateException: Received message from unsupported version: [35.24.79-beta2] minimal compatible version is: [6.7.0]
at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:139) ~[?:?]
at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:115) ~[?:?]
at org.elasticsearch.transport.TcpTransport$1.doRun(TcpTransport.java:975) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[?:?]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[?:?]
at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:967) ~[?:?]
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:942) ~[?:?]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[?:?]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
at java.lang.Thread.run(Thread.java:835) ~[?:?]
To Reproduce Steps to reproduce the behavior:
- create elastic oss cluster v7.0
- create new os 1.3.2 node and try to join it to the existing es cluster.
Expected behavior the os node would be expect to joing the es cluster such that i can move migrate data to it.
Plugins opensearch-alerting opensearch-anomaly-detection opensearch-asynchronous-search opensearch-cross-cluster-replication opensearch-index-management opensearch-job-scheduler opensearch-knn opensearch-ml opensearch-observability opensearch-reports-scheduler opensearch-sql repository-s3
Host/Environment (please complete the following information):
- OS: debian
- Version 11.2
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 17 (8 by maintainers)
@akomlik See https://forum.opensearch.org/t/open-distro-artifacts/10493/3, drop a note to opensearch at amazon dot com and we’ll help you locate those artifacts to enable the upgrade path.