pulsar: Pulsar functions worker fails to initialize
Describe the bug The Pulsar functions worker fails to initialize. This breaks Pulsar Functions functionality.
19:41:53.512 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught
java.lang.ArrayIndexOutOfBoundsException: -2
at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:60) ~[com.google.guava-guava-21.0.jar:?]
at org.apache.bookkeeper.client.PendingReadOp$SequenceReadRequest.sendNextRead(PendingReadOp.java:400) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.client.PendingReadOp$SequenceReadRequest.read(PendingReadOp.java:382) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.client.PendingReadOp.initiate(PendingReadOp.java:526) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.client.LedgerRecoveryOp.doRecoveryRead(LedgerRecoveryOp.java:148) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.client.LedgerRecoveryOp.access$000(LedgerRecoveryOp.java:37) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.client.LedgerRecoveryOp$1.readLastConfirmedDataComplete(LedgerRecoveryOp.java:109) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.client.ReadLastConfirmedOp.readEntryComplete(ReadLastConfirmedOp.java:135) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion$1.readEntryComplete(PerChannelBookieClient.java:1797) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion.handleReadResponse(PerChannelBookieClient.java:1878) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion.handleV2Response(PerChannelBookieClient.java:1831) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.safeRun(PerChannelBookieClient.java:1321) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.9.0.jar:4.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
**root@pulsar-broker-56d59cf97d-zwtxf:/pulsar# bin/pulsar-admin functions list
Function worker service is not done initializing. Please try again in a little while.
Reason: HTTP 503 Service Unavailable
Note: this issue persisted for a time period on the scale of hours until I gave up and tore down the cluster.
To Reproduce Reproduction steps unclear. Could possibly be related to updating existing functions.
Expected behavior Expect Pulsar functions to behave as expected - pulsar-admin functions list should not say there’s an HTTP 503, etc.
Platform (please complete the following information):
- Using the pulsar-all image tagged 2.3.0.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 2
- Comments: 17 (11 by maintainers)
Commits related to this issue
- Disable sticky read by default *Motivation* 2.3.x releases turn on StickyRead by default. But StickyRead causes a lot of ArrayIndexOutOfBoundException. See: apache/bookkeeper#1970 and apache/pulsar... — committed to sijie/pulsar by sijie 5 years ago
- Disable sticky read by default (#4526) *Motivation* 2.3.x releases turn on StickyRead by default. But StickyRead causes a lot of ArrayIndexOutOfBoundException. See: apache/bookkeeper#1970 and ... — committed to apache/pulsar by sijie 5 years ago
- Issue #1970: Ensure getStickyReadBookieIndex returns valid bookie index Descriptions of the changes in this PR: Master Issue: #1970 Related Issues: apache/pulsar#3715 apache/pulsar#4526 *Motiv... — committed to apache/bookkeeper by sijie 5 years ago
- Issue #1970: Ensure getStickyReadBookieIndex returns valid bookie index Descriptions of the changes in this PR: Master Issue: #1970 Related Issues: apache/pulsar#3715 apache/pulsar#4526 *Motiv... — committed to apache/bookkeeper by sijie 5 years ago
The problem was due to the StickyRead feature introduced in bookkeeper since 4.9.0. Pulsar 2.3.0 turns on this feature by default.
I am fixing this issue. At the same time, for people who encountered this issue, you can disable sticky read on broker to get around it.
Closed this issue since the bookkeeper version is updated to 4.10 including the bookkeeper client fix.