prometheus: Prometheus hangs indefinitely when exiting gracefully

What did you do?

For initial context, we run Prometheus within Kubernetes which is all managed by Prometheus Operator. When we make a change that requires the Prometheus pods to restart (ie: updating a Prometheus resource), we will see each Prometheus pod enter a Terminating state, and it will remain in that state for 10 minutes. After that time, the pod is killed and the restart happens as expected. The 10-minute period is due to the hard-coded teminationGracePeriod set within the StatefulSet by the Prometheus Operator, as seen here. We have been unable to figure out why these pods are taking this long to terminate.

What did you expect to see?

When a pod restarts “normally” (which does happen, but it’s very rare for us), we see the log lines mentioned below (in the “what did you see instead” section), but with one additional log line added at the bottom:

level=info ts=2020-02-03T20:10:35.880Z caller=main.go:730 msg="See you next time!"

Also, the pod actually terminates successfully after a second or two from receiving a SIGTERM signal.

What did you see instead? Under which circumstances?

As soon as the pods receive a SIGTERM from Kubernetes, we see the following logs:

level=warn ts=2020-02-03T19:43:02.640Z caller=main.go:494 msg="Received SIGTERM, exiting gracefully..."
level=info ts=2020-02-03T19:43:02.640Z caller=main.go:517 msg="Stopping scrape discovery manager..."
level=info ts=2020-02-03T19:43:02.640Z caller=main.go:531 msg="Stopping notify discovery manager..."
level=info ts=2020-02-03T19:43:02.640Z caller=main.go:553 msg="Stopping scrape manager..."
level=info ts=2020-02-03T19:43:02.640Z caller=main.go:513 msg="Scrape discovery manager stopped"
level=info ts=2020-02-03T19:43:02.640Z caller=main.go:527 msg="Notify discovery manager stopped"
level=info ts=2020-02-03T19:43:02.640Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..."
level=info ts=2020-02-03T19:43:02.640Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped"
level=info ts=2020-02-03T19:43:02.640Z caller=main.go:547 msg="Scrape manager stopped"
level=info ts=2020-02-03T19:43:02.648Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..."
level=info ts=2020-02-03T19:43:02.648Z caller=main.go:718 msg="Notifier manager stopped"

No other log lines are logged and the pod is eventually killed after ten minutes. Note that we don’t see the “see you next time” log line at all in this scenario.

Environment

We run Prometheus within Kubernetes, using Prometheus Operator to manage the Prometheus pods.

Version Info:

• Kubernetes: v1.13.10
• Prometheus: v2.15.2
• Prometheus Operator: v0.32

Each Prometheus pod uses a Ceph mount (managed by Kubernetes) for Prometheus data.

Anyone have any ideas? I did try enabling debug logs and did not notice any additional log lines during the shutdown process (post SIGTERM being received). Furthermore, I found issue #5397 which seems very similar, and it seems like they were waiting for systemd to kill the process due to a timeout. I don’t have any leads at the moment though other than my gut feeling that tells me this is likely something related to the TSDB / ceph volumes. For what it’s worth, I did not find any single errors in the kubelet logs either.