charts: Graceful shutdown is not working without adding sleep in preStop
Graceful shutdown is not working because API call to endpoint - http://localhost:8080/v1/info/state returns 200 response instantly and kubernetes is killing that pod without waiting for terminationGracePeriodSeconds. While preStop command should wait for active tasks to be completed. Currently there can be 2 workarounds
- Add sleep time of
Values.worker.terminationGracePeriodSecondsin preStop hook - API call should return 200 response only when all active tasks are completed rather than scheduling them in ScheduleFuture.
Related PR for graceful shutdown - https://github.com/trinodb/charts/pull/226/files#diff-d18e49ecf4264ad889f2d0a897b2075c8c99b542f69e2b91106a8d1809c0b386R190-R202
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 21 (10 by maintainers)
Does adding the sleep mean the container will always hang around for the full period regardless of whether the trino process has shut down already?
Is there something we could do in the prestop to watch for the existence of the trino process?
Maybe tail --pid=1 -f /dev/null
That should end when the trino process finishes I think
https://unix.stackexchange.com/questions/427115/listen-for-exit-of-process-given-pid
@lozbrown I like your suggestion a lot!
My observations and logs are listed above. Since then, my fix has been to expand locally and modify the helm chart template to add “sleep 240” at the end of the termination prestop command. THis is a no-brainer.
I hope this issue will be addressed in the helm chart (at least until the root cause is found). A simple fix could be the sleep command.
I have the following values for the worker.
Now, if I exec into the container and run:
After running the above command, the logs appear as follows:
These logs are as expected—about one minute after the
Shutdown requestedlog, we seeWaiting for 0 active tasks to finish, and after another minute, the container terminates.However, if the worker deployment is restarted, the logs are as follows:
The JVM shuts down immediately. Additionally, I noticed something that might help—containers enter an error state before they terminate. It seems that when the
preStopcommand runs, it fails, causing the container to terminate immediately. @justrelax19 @jarlva Does this error state of container happen to you as well?@nineinchnick, please note my result as well as @justrelax19 above. Adding the
sleep XXat the end of the curl command fixed the issue.The solution, which we verified, is to add the sleep command to the helm chart.
And the above test is just checking if we are getting the logline
Shutdown requestedor not at the time of deletion of pod which is not correct because after this logline, the actual invocation ofwaitForActiveTasksis called which is not being checked by the above test. Therefore, if we will modify and wait for the loglinelog.info("Waiting for " + activeTasks.size() + " active tasks to finish");written in io.trino.server.NodeStateManager#waitActiveTasksToFinish, then we might be able to reproduce the issue.+1 to what @jarlva said. We added sleep equal to the
terminationGracePeriodPerSecondsat the end of preStop cycle in the curl command and then theshutdownmethod in the [shutdownHandler.schedule](java.util.concurrent.ScheduledExecutorService#schedule(java.lang.Runnable, long, java.util.concurrent.TimeUnit)) got invoked and it waited for the active tasks to finish.Here is the updated
preStophook which we are using that is enabling us to invoke thewaitForActiveTasksToFinishmethodAlso @nineinchnick , you can refer to the logs @jarlva attached to see that the LifeCycleManager simply shutdown the process within few seconds