trino: Graceful shutdown may hang indefinitely if node crashed

I’m using Trino 406, configured with 30s as the shutdown.grace-period.

One of our nodes crashed with a “java.lang.OutOfMemoryError: Java heap space” error. We restarted it and continued using this worker normally on the cluster until, a couple hours later, we decided to terminate it. We sent a graceful shutdown request to that node and confirmed it had switched into the SHUTTING_DOWN state, but after 7 hours it never actually terminated the Trino process, so we were unable to terminate it gracefully.

From the worker I was able to confirm that it was still on the SHUTTING_DOWN state and that there were no active queries being run by it:

$ curl localhost:8080/v1/info/state
"SHUTTING_DOWN"

$ curl localhost:8080/v1/status
{"nodeId":"presto320","nodeVersion":{"version":"406"},"environment":"production","coordinator":false,"uptime":"10.55h","externalAddress":"x","internalAddress":"x","memoryInfo":{"availableProcessors":64,"pool":{"maxBytes":345744867328,"reservedBytes":0,"reservedRevocableBytes":0,"queryMemoryReservations":{},"queryMemoryAllocations":{},"queryMemoryRevocableReservations":{},"taskMemoryReservations":{},"taskMemoryRevocableReservations":{},"freeBytes":345744867328}},"processors":64,"processCpuLoad":0.011531810534497226,"systemCpuLoad":0.011366032123771604,"heapUsed":243704836488,"heapAvailable":493921239040,"nonHeapUsed":755459576}

Though from this worker logs I could see a single kind of HTTP request being repeated many times over non-stop:

2023-07-18T14:18:46.145Z        <coordinator IP>    GET     /v1/task/20230718_032419_12112_it5df.57.2.0/dynamicfilters      null    null    200     0       40      1       e0f265b13b4e4ed58fd575b060a75b4a0006d3f42d      HTTP/1.1        0       0       0       null
2023-07-18T14:18:46.146Z        <coordinator IP>    GET     /v1/task/20230718_032419_12112_it5df.57.2.0/dynamicfilters      null    null    200     0       40      0       e0f265b13b4e4ed58fd575b060a75b4a0006d3f42e      HTTP/1.1        0       0       0       null
2023-07-18T14:18:46.146Z        <coordinator IP>    GET     /v1/task/20230718_032419_12112_it5df.57.2.0/dynamicfilters      null    null    200     0       40      0       e0f265b13b4e4ed58fd575b060a75b4a0006d3f42f      HTTP/1.1        0       0       0       null

This contains the query id of a query that failed when this node had crashed due to the OOM issue, and this seems to be blocking the node’s shutdown, even though the query is not running anymore.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 18 (13 by maintainers)

Most upvoted comments

I think I know what’s happening:

  1. coordinator receives some information about new version of DF on worker
  2. worker fails with OOM
  3. worker is restarted
  4. DynamicFiltersCollector keeps pinging new worker for DF on dead task, thus keeping the task not done nor abandoned