charts: Graceful shutdown is not working without adding sleep in preStop

Graceful shutdown is not working because API call to endpoint - http://localhost:8080/v1/info/state returns 200 response instantly and kubernetes is killing that pod without waiting for terminationGracePeriodSeconds. While preStop command should wait for active tasks to be completed. Currently there can be 2 workarounds

  1. Add sleep time of Values.worker.terminationGracePeriodSeconds in preStop hook
  2. API call should return 200 response only when all active tasks are completed rather than scheduling them in ScheduleFuture.

Related PR for graceful shutdown - https://github.com/trinodb/charts/pull/226/files#diff-d18e49ecf4264ad889f2d0a897b2075c8c99b542f69e2b91106a8d1809c0b386R190-R202

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 21 (10 by maintainers)

Most upvoted comments

Does adding the sleep mean the container will always hang around for the full period regardless of whether the trino process has shut down already?

Is there something we could do in the prestop to watch for the existence of the trino process?

Maybe tail --pid=1 -f /dev/null

That should end when the trino process finishes I think

https://unix.stackexchange.com/questions/427115/listen-for-exit-of-process-given-pid

@lozbrown I like your suggestion a lot!

My observations and logs are listed above. Since then, my fix has been to expand locally and modify the helm chart template to add “sleep 240” at the end of the termination prestop command. THis is a no-brainer.

I hope this issue will be addressed in the helm chart (at least until the root cause is found). A simple fix could be the sleep command.

I have the following values for the worker.

worker:
  terminationGracePeriodSeconds: 120
  gracefulShutdown:
    enabled: true
    gracePeriodSeconds: 60

Now, if I exec into the container and run:

curl -v -X PUT -d '"SHUTTING_DOWN"' -H 'Content-type: application/json' -H 'X-Trino-User: admin' http://localhost:8080/v1/info/state

After running the above command, the logs appear as follows:

2025-03-20T10:17:07.039Z    INFO    http-worker-78    io.trino.server.ServerInfoResource    Worker State change requested: ACTIVE -> SHUTTING_DOWN
2025-03-20T10:17:07.040Z    INFO    http-worker-78    io.trino.server.NodeStateManager    Shutdown requested
2025-03-20T10:18:07.041Z    INFO    shutdown-handler-0    io.trino.server.NodeStateManager    Waiting for 0 active tasks to finish
2025-03-20T10:19:07.044Z    INFO    lifecycle-stopper-0    org.eclipse.jetty.server.Server    Stopped oejs.Server@590dea35{STOPPING}[12.0.16,sto=0]
2025-03-20T10:19:07.047Z    INFO    lifecycle-stopper-0    org.eclipse.jetty.server.AbstractConnector    Stopped http@61836cd9{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8080}
2025-03-20T10:19:07.053Z    INFO    lifecycle-stopper-0    org.eclipse.jetty.ee10.servlet.ServletContextHandler    Stopped oeje10s.ServletContextHandler@26d7cb0d{ROOT,/,b=null,a=AVAILABLE,vh=[@https,@http],h=GzipHa
ndler@878feb2{STOPPED,min=32,inflate=-1}}
2025-03-20T10:19:09.159Z    INFO    LifeCycleManager Shutdown Hook    io.airlift.bootstrap.LifeCycleManager    JVM is shutting down, cleaning up
2025-03-20T10:19:09.159Z    INFO    LifeCycleManager Shutdown Hook    io.airlift.bootstrap.LifeCycleManager    JVM is shutting down, cleaning **up**

These logs are as expected—about one minute after the Shutdown requested log, we see Waiting for 0 active tasks to finish, and after another minute, the container terminates.

However, if the worker deployment is restarted, the logs are as follows:

2025-03-20T11:32:33.982Z    INFO    http-worker-75    io.trino.server.ServerInfoResource    Worker State change requested: ACTIVE -> SHUTTING_DOWN
2025-03-20T11:32:33.982Z    INFO    http-worker-75    io.trino.server.NodeStateManager    Shutdown requested
2025-03-20T11:32:33.991Z    INFO    LifeCycleManager Shutdown Hook    io.airlift.bootstrap.LifeCycleManager    JVM is shutting down, cleaning up
2025-03-20T11:32:33.992Z    INFO    LifeCycleManager Shutdown Hook    io.airlift.bootstrap.LifeCycleManager    JVM is shutting down, cleaning up
2025-03-20T11:32:33.992Z    INFO    LifeCycleManager Shutdown Hook    io.airlift.bootstrap.LifeCycleManager    JVM is shutting down, cleaning up
2025-03-20T11:32:33.992Z    INFO    LifeCycleManager Shutdown Hook    io.airlift.bootstrap.LifeCycleManager    JVM is shutting down, cleaning up
2025-03-20T11:32:33.992Z    INFO    LifeCycleManager Shutdown Hook    io.airlift.bootstrap.LifeCycleManager    JVM is shutting down, cleaning up
2025-03-20T11:32:33.992Z    INFO    LifeCycleManager Shutdown Hook    io.airlift.bootstrap.LifeCycleManager    JVM is shutting down, cleaning up
2025-03-20T11:32:33.992Z    INFO    LifeCycleManager Shutdown Hook    io.airlift.bootstrap.LifeCycleManager    JVM is shutting down, cleaning up

The JVM shuts down immediately. Additionally, I noticed something that might help—containers enter an error state before they terminate. It seems that when the preStop command runs, it fails, causing the container to terminate immediately. @justrelax19 @jarlva Does this error state of container happen to you as well?

Image

@nineinchnick, please note my result as well as @justrelax19 above. Adding the sleep XX at the end of the curl command fixed the issue.

The solution, which we verified, is to add the sleep command to the helm chart.

I don’t fully understand the problem, you’re saying after the current preStop hook sends a request to /v1/info/state, the worker shuts down immediately, interrupting running queries?

There’s a test for graceful shutdown - https://github.com/trinodb/charts/blob/main/tests/trino/test-graceful-shutdown-values.yaml and https://github.com/trinodb/charts/blob/main/charts/trino/templates/tests/test-graceful-sh utdown.yaml Does it need to be modified to reproduce this issue?

And the above test is just checking if we are getting the logline Shutdown requested or not at the time of deletion of pod which is not correct because after this logline, the actual invocation of waitForActiveTasks is called which is not being checked by the above test. Therefore, if we will modify and wait for the logline log.info("Waiting for " + activeTasks.size() + " active tasks to finish"); written in io.trino.server.NodeStateManager#waitActiveTasksToFinish, then we might be able to reproduce the issue.

+1 to what @jarlva said. We added sleep equal to the terminationGracePeriodPerSeconds at the end of preStop cycle in the curl command and then the shutdown method in the [shutdownHandler.schedule](java.util.concurrent.ScheduledExecutorService#schedule(java.lang.Runnable, long, java.util.concurrent.TimeUnit)) got invoked and it waited for the active tasks to finish.

Here is the updated preStop hook which we are using that is enabling us to invoke the waitForActiveTasksToFinish method

            {{- else if .Values.worker.gracefulShutdown.enabled }}
            preStop:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - >-
                    curl -v -X PUT
                    -d '"SHUTTING_DOWN"'
                    -H 'Content-type: application/json'
                    -H 'X-Trino-User: admin'
                    http://localhost:{{- .Values.service.port -}}/v1/info/state; 
                    sleep {{ .Values.worker.terminationGracePeriodSeconds }}
            {{- end }}

Also @nineinchnick , you can refer to the logs @jarlva attached to see that the LifeCycleManager simply shutdown the process within few seconds