nats.java: ArrayIndexOutOfBoundsException on attempt to reconnect
Defect
We observed one of our Nats clients in our production enviroment flooded log with the following exceptions.
- First 40 minutes it was throwing the following exception during publish() call (messages consumption was healthy):
java.lang.IllegalStateException: Output queue is full 5000
at io.nats.client.impl.MessageQueue.push(MessageQueue.java:124)
at io.nats.client.impl.MessageQueue.push(MessageQueue.java:110)
at io.nats.client.impl.NatsConnectionWriter.queue(NatsConnectionWriter.java:209)
at io.nats.client.impl.NatsConnection.queueOutgoing(NatsConnection.java:1353)
at io.nats.client.impl.NatsConnection.publishInternal(NatsConnection.java:765)
at io.nats.client.impl.NatsConnection.publish(NatsConnection.java:733)
- Then both publishing and consumption didn’t work for another 40 minutes till service was terminated. At this time log was flooded with the following exceptions (each time with the same last destination index and byte index):
java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination index 175759 out of bounds for byte[131555]
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
at io.nats.client.impl.NatsConnection.tryToConnect(NatsConnection.java:380)
at io.nats.client.impl.NatsConnection.reconnect(NatsConnection.java:254)
at io.nats.client.impl.NatsConnection.closeSocket(NatsConnection.java:583)
at io.nats.client.impl.NatsConnection.lambda$handleCommunicationIssue$3(NatsConnection.java:541)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination index 175759 out of bounds for byte[131555]
at java.base/java.lang.System.arraycopy(Native Method)
at io.nats.client.impl.NatsConnectionWriter.sendMessageBatch(NatsConnectionWriter.java:147)
at io.nats.client.impl.NatsConnectionWriter.run(NatsConnectionWriter.java:188)
... 5 common frames omitted
Our client configuration:
Options options = new Options.Builder()
.servers(natsBootstrapServers.addresses)
.errorListener(errorListener)
.connectionTimeout(Duration.ofSeconds(5))
.maxReconnects(-1) // -1 = Infinite
.connectionName(getHostName())
.pingInterval(Duration.ofSeconds(2))
.build();
We don’t have a reproduction scenario yet.
Versions of io.nats:jnats
and nats-server
:
nats server v 2.7.4 jnats v 2.14.1
OS/Container environment:
adoptopenjdk/openjdk11:jdk-11.0.8_10-alpine
Steps or code to reproduce the issue:
Expected result:
Actual result:
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 19 (7 by maintainers)
Ok I think I see it now. We advance the msg to the next, but never send the one that would have been too big for the buffer. It probably doesn’t happen much because of how often we run this code and the default buffer is plenty big, but on cases where the buffer was small or the message was really big, this can happen a lot.