pulsar-helm-chart: Brokers fail starting on GKE Autopilot

Describe the bug

I’m installing the helm on GKE with Autopilot turned on but my brokers fail starting up. If I turn off functions by setting components.functions: false the brokers start up correctly.

This is the exception on my brokers’ initialization:

2023-12-19T00:47:53,609+0000 [main] INFO  org.apache.pulsar.functions.utils.functions.FunctionUtils - Searching for functions in /pulsar/./functions
2023-12-19T00:47:53,610+0000 [main] WARN  org.apache.pulsar.functions.utils.functions.FunctionUtils - Functions archive directory not found
2023-12-19T00:47:53,612+0000 [main] ERROR org.apache.pulsar.functions.worker.PulsarWorkerService - Error Starting up in worker
org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
    at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:251) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:301) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:457) ~[org.apache.pulsar-pulsar-functions-worker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1608) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:795) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
Caused by: java.lang.IllegalStateException: Client instance has been closed.
    at org.glassfish.jersey.internal.guava.Preconditions.checkState(Preconditions.java:169) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyClient.checkNotClosed(JerseyClient.java:257) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.checkNotClosed(JerseyWebTarget.java:118) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:129) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:38) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.topicPath(TopicsImpl.java:1319) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptionsAsync(TopicsImpl.java:619) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.lambda$getSubscriptions$20(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:292) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    ... 6 more
2023-12-19T00:47:53,619+0000 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:585) ~[org.apache.pulsar-pulsar-functions-worker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1608) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:795) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
Caused by: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
    at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:251) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:301) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:457) ~[org.apache.pulsar-pulsar-functions-worker-2.10.2.jar:2.10.2]
    ... 4 more
Caused by: java.lang.IllegalStateException: Client instance has been closed.
    at org.glassfish.jersey.internal.guava.Preconditions.checkState(Preconditions.java:169) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyClient.checkNotClosed(JerseyClient.java:257) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.checkNotClosed(JerseyWebTarget.java:118) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:129) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:38) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.topicPath(TopicsImpl.java:1319) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptionsAsync(TopicsImpl.java:619) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.lambda$getSubscriptions$20(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:292) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:457) ~[org.apache.pulsar-pulsar-functions-worker-2.10.2.jar:2.10.2]
    ... 4 more
2023-12-19T00:47:53,620+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.broker.PulsarService - Closing PulsarService
2023-12-19T00:47:53,621+0000 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
org.apache.pulsar.broker.PulsarServerException: java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:827) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
Caused by: java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:585) ~[org.apache.pulsar-pulsar-functions-worker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1608) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:795) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    ... 2 more
Caused by: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
    at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:251) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:301) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:457) ~[org.apache.pulsar-pulsar-functions-worker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1608) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:795) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    ... 2 more
Caused by: java.lang.IllegalStateException: Client instance has been closed.
    at org.glassfish.jersey.internal.guava.Preconditions.checkState(Preconditions.java:169) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyClient.checkNotClosed(JerseyClient.java:257) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.checkNotClosed(JerseyWebTarget.java:118) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:129) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:38) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.topicPath(TopicsImpl.java:1319) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptionsAsync(TopicsImpl.java:619) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.lambda$getSubscriptions$20(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:292) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:457) ~[org.apache.pulsar-pulsar-functions-worker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1608) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:795) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    ... 2 more

Any ideas on what may be causing this?

To Reproduce Steps to reproduce the behavior:

  1. Create a GKE Cluster on Autopilot
  2. Install helm
  3. See errors in brokers

Expected behavior I expected the helm chart to install the cluster correctly with no errors.

About this issue

  • Original URL
  • State: open
  • Created 6 months ago
  • Comments: 15 (10 by maintainers)

Most upvoted comments

Understood, I copied and posted the logs before updating to chart 3.1.0. After updating the issue persisted, I just didn’t copy the log messages after that because they were exactly the same.