thanos: receive: does not cleanup old data on disk, but tries to load it on boot, causing OOM down the line

Thanos, Prometheus and Golang version used: quay.io/thanos/thanos:v0.21.1

Object Storage Provider: GCS

What happened: Running with:

      --tsdb.retention=1h
      --tsdb.path=/thanos-data
      --tsdb.allow-overlapping-blocks
      --shipper.allow-out-of-order-uploads 

Several days old data is left on disk, with no attempts of shipping it to storage. Not necessarily days worth of data, tho I have not really checked that.

Some WAL segments are left forever on disk, eventually causing OOM on boot, as memory needs becomes ever growing. Giving it enough memory to load all segments appears to make no difference, as even when loaded no attempts to ship them are made, even after several days.

Manual removal of the affected segments on disk appears to resolve it, as nothing is logged about data inconsistency after doing so.

What you expected to happen:

Old data is removed from disk and not attempted to be loaded forever.

How to reproduce it (as minimally and precisely as possible): Run a receiver and ingest data to it, observe data being left on disk. It may need to run for a few days, and restart a few times, maybe be shutdown for sometime in-between too, not entirely sure.

Full logs to relevant components:

Looking in the WAL directory (captured a few weeks ago when first encountering it)

/ # ls -lah /thanos-data/default-tenant/wal/
total 127M
drwxr-xr-x  104 root     root       12.0K Jun 18 09:05 .
drwxr-xr-x    6 root     root        4.0K Jun 18 09:05 ..
-rw-r--r--    1 root     root        3.2M Jun 15 09:12 00000739
-rw-r--r--    1 root     root       96.0K Jun 15 11:16 00000740
-rw-r--r--    1 root     root       10.5K Jun 15 11:16 00000741
-rw-r--r--    1 root     root           0 Jun 15 13:03 00000742
-rw-r--r--    1 root     root           0 Jun 15 13:11 00000743
-rw-r--r--    1 root     root       64.0K Jun 15 15:15 00000744
-rw-r--r--    1 root     root       15.3M Jun 15 15:20 00000745
-rw-r--r--    1 root     root        4.6M Jun 15 15:20 00000746
-rw-r--r--    1 root     root       29.4M Jun 15 16:12 00000747
-rw-r--r--    1 root     root        7.6M Jun 15 16:13 00000748
-rw-r--r--    1 root     root           0 Jun 15 17:01 00000749
-rw-r--r--    1 root     root       15.3M Jun 15 17:26 00000750
-rw-r--r--    1 root     root           0 Jun 15 17:26 00000751
-rw-r--r--    1 root     root       45.3M Jun 15 19:20 00000752
-rw-r--r--    1 root     root        5.6M Jun 15 19:20 00000753
-rw-r--r--    1 root     root           0 Jun 15 21:05 00000754
-rw-r--r--    1 root     root           0 Jun 15 21:07 00000755
-rw-r--r--    1 root     root           0 Jun 15 23:06 00000756
-rw-r--r--    1 root     root           0 Jun 16 01:05 00000757
-rw-r--r--    1 root     root           0 Jun 16 01:07 00000758
-rw-r--r--    1 root     root           0 Jun 16 03:06 00000759
-rw-r--r--    1 root     root           0 Jun 16 03:07 00000760
-rw-r--r--    1 root     root           0 Jun 16 05:05 00000761
-rw-r--r--    1 root     root           0 Jun 16 06:49 00000762
-rw-r--r--    1 root     root           0 Jun 16 06:50 00000763
-rw-r--r--    1 root     root           0 Jun 16 06:52 00000764
-rw-r--r--    1 root     root           0 Jun 16 07:03 00000765
-rw-r--r--    1 root     root           0 Jun 16 07:05 00000766
-rw-r--r--    1 root     root           0 Jun 16 07:07 00000767
-rw-r--r--    1 root     root           0 Jun 16 09:03 00000768
-rw-r--r--    1 root     root           0 Jun 16 09:04 00000769
-rw-r--r--    1 root     root           0 Jun 16 09:06 00000770
-rw-r--r--    1 root     root           0 Jun 16 09:08 00000771
-rw-r--r--    1 root     root           0 Jun 16 09:10 00000772
-rw-r--r--    1 root     root           0 Jun 16 09:13 00000773
-rw-r--r--    1 root     root           0 Jun 16 09:17 00000774
-rw-r--r--    1 root     root           0 Jun 16 09:24 00000775
-rw-r--r--    1 root     root           0 Jun 16 10:23 00000776
-rw-r--r--    1 root     root           0 Jun 16 10:25 00000777
-rw-r--r--    1 root     root           0 Jun 16 10:27 00000778
-rw-r--r--    1 root     root           0 Jun 16 10:28 00000779
-rw-r--r--    1 root     root           0 Jun 16 10:31 00000780
-rw-r--r--    1 root     root           0 Jun 16 10:34 00000781
-rw-r--r--    1 root     root           0 Jun 16 10:38 00000782
-rw-r--r--    1 root     root           0 Jun 16 10:44 00000783
-rw-r--r--    1 root     root           0 Jun 16 10:51 00000784
-rw-r--r--    1 root     root           0 Jun 16 10:57 00000785
-rw-r--r--    1 root     root           0 Jun 16 11:04 00000786
-rw-r--r--    1 root     root           0 Jun 16 11:11 00000787
-rw-r--r--    1 root     root           0 Jun 16 11:17 00000788
-rw-r--r--    1 root     root           0 Jun 16 11:24 00000789
-rw-r--r--    1 root     root           0 Jun 16 11:30 00000790
-rw-r--r--    1 root     root           0 Jun 16 11:37 00000791
-rw-r--r--    1 root     root           0 Jun 16 11:54 00000792
-rw-r--r--    1 root     root           0 Jun 16 11:55 00000793
-rw-r--r--    1 root     root           0 Jun 16 11:57 00000794
-rw-r--r--    1 root     root           0 Jun 16 11:59 00000795
-rw-r--r--    1 root     root           0 Jun 16 12:01 00000796
-rw-r--r--    1 root     root           0 Jun 16 12:03 00000797
-rw-r--r--    1 root     root           0 Jun 16 12:07 00000798
-rw-r--r--    1 root     root           0 Jun 16 12:14 00000799
-rw-r--r--    1 root     root           0 Jun 16 12:21 00000800
-rw-r--r--    1 root     root           0 Jun 16 12:27 00000801
-rw-r--r--    1 root     root           0 Jun 16 12:33 00000802
-rw-r--r--    1 root     root           0 Jun 16 12:34 00000803
-rw-r--r--    1 root     root           0 Jun 16 12:36 00000804
-rw-r--r--    1 root     root           0 Jun 16 12:38 00000805
-rw-r--r--    1 root     root           0 Jun 16 12:40 00000806
-rw-r--r--    1 root     root           0 Jun 16 12:43 00000807
-rw-r--r--    1 root     root           0 Jun 16 12:48 00000808
-rw-r--r--    1 root     root           0 Jun 16 12:54 00000809
-rw-r--r--    1 root     root           0 Jun 16 13:01 00000810
-rw-r--r--    1 root     root           0 Jun 16 13:07 00000811
-rw-r--r--    1 root     root           0 Jun 16 13:14 00000812
-rw-r--r--    1 root     root           0 Jun 16 13:20 00000813
-rw-r--r--    1 root     root           0 Jun 16 13:27 00000814
-rw-r--r--    1 root     root           0 Jun 16 14:38 00000815
-rw-r--r--    1 root     root           0 Jun 16 14:39 00000816
-rw-r--r--    1 root     root           0 Jun 16 14:41 00000817
-rw-r--r--    1 root     root           0 Jun 16 14:43 00000818
-rw-r--r--    1 root     root           0 Jun 16 14:45 00000819
-rw-r--r--    1 root     root           0 Jun 16 14:48 00000820
-rw-r--r--    1 root     root           0 Jun 16 14:52 00000821
-rw-r--r--    1 root     root           0 Jun 16 14:59 00000822
-rw-r--r--    1 root     root           0 Jun 16 15:05 00000823
-rw-r--r--    1 root     root           0 Jun 16 15:12 00000824
-rw-r--r--    1 root     root           0 Jun 16 15:18 00000825
-rw-r--r--    1 root     root           0 Jun 16 16:40 00000826
-rw-r--r--    1 root     root           0 Jun 16 16:42 00000827
-rw-r--r--    1 root     root           0 Jun 16 16:43 00000828
-rw-r--r--    1 root     root           0 Jun 16 16:45 00000829
-rw-r--r--    1 root     root           0 Jun 16 16:47 00000830
-rw-r--r--    1 root     root           0 Jun 16 16:50 00000831
-rw-r--r--    1 root     root           0 Jun 16 16:54 00000832
-rw-r--r--    1 root     root           0 Jun 16 17:01 00000833
-rw-r--r--    1 root     root           0 Jun 16 17:07 00000834
-rw-r--r--    1 root     root           0 Jun 16 19:04 00000835
-rw-r--r--    1 root     root           0 Jun 16 19:05 00000836
-rw-r--r--    1 root     root           0 Jun 16 19:07 00000837
-rw-r--r--    1 root     root           0 Jun 16 19:09 00000838
-rw-r--r--    1 root     root           0 Jun 16 19:11 00000839
-rw-r--r--    1 root     root           0 Jun 16 19:14 00000840
-rw-r--r--    1 root     root           0 Jun 16 19:18 00000841
-rw-r--r--    1 root     root           0 Jun 16 19:24 00000842
-rw-r--r--    1 root     root           0 Jun 16 21:02 00000843
-rw-r--r--    1 root     root           0 Jun 16 21:04 00000844
-rw-r--r--    1 root     root           0 Jun 16 21:06 00000845
-rw-r--r--    1 root     root           0 Jun 16 23:03 00000846
-rw-r--r--    1 root     root           0 Jun 16 23:05 00000847
-rw-r--r--    1 root     root           0 Jun 16 23:07 00000848
-rw-r--r--    1 root     root           0 Jun 16 23:09 00000849
-rw-r--r--    1 root     root           0 Jun 17 01:06 00000850
-rw-r--r--    1 root     root           0 Jun 17 01:07 00000851
-rw-r--r--    1 root     root           0 Jun 17 03:05 00000852
-rw-r--r--    1 root     root           0 Jun 17 03:07 00000853
-rw-r--r--    1 root     root           0 Jun 17 03:08 00000854
-rw-r--r--    1 root     root           0 Jun 17 05:06 00000855
-rw-r--r--    1 root     root           0 Jun 17 05:07 00000856
-rw-r--r--    1 root     root           0 Jun 17 07:01 00000857
-rw-r--r--    1 root     root           0 Jun 17 07:03 00000858
-rw-r--r--    1 root     root           0 Jun 17 07:05 00000859
-rw-r--r--    1 root     root           0 Jun 17 07:08 00000860
-rw-r--r--    1 root     root           0 Jun 17 07:10 00000861
-rw-r--r--    1 root     root           0 Jun 17 07:26 00000862
-rw-r--r--    1 root     root           0 Jun 17 07:28 00000863
-rw-r--r--    1 root     root           0 Jun 17 07:29 00000864
-rw-r--r--    1 root     root           0 Jun 17 07:31 00000865
-rw-r--r--    1 root     root           0 Jun 17 09:01 00000866
-rw-r--r--    1 root     root           0 Jun 17 09:03 00000867
-rw-r--r--    1 root     root           0 Jun 17 09:04 00000868
-rw-r--r--    1 root     root           0 Jun 17 09:06 00000869
-rw-r--r--    1 root     root           0 Jun 17 09:08 00000870
-rw-r--r--    1 root     root           0 Jun 17 09:11 00000871
-rw-r--r--    1 root     root           0 Jun 17 09:15 00000872
-rw-r--r--    1 root     root           0 Jun 17 09:21 00000873
-rw-r--r--    1 root     root           0 Jun 17 09:28 00000874
-rw-r--r--    1 root     root           0 Jun 17 09:34 00000875
-rw-r--r--    1 root     root           0 Jun 17 09:41 00000876
-rw-r--r--    1 root     root           0 Jun 17 09:47 00000877
-rw-r--r--    1 root     root           0 Jun 17 09:53 00000878
-rw-r--r--    1 root     root           0 Jun 17 10:00 00000879
-rw-r--r--    1 root     root           0 Jun 17 10:06 00000880
-rw-r--r--    1 root     root           0 Jun 17 10:13 00000881
-rw-r--r--    1 root     root           0 Jun 17 10:19 00000882
-rw-r--r--    1 root     root           0 Jun 17 10:26 00000883
-rw-r--r--    1 root     root           0 Jun 17 10:32 00000884
-rw-r--r--    1 root     root           0 Jun 17 10:39 00000885
-rw-r--r--    1 root     root           0 Jun 17 10:45 00000886
-rw-r--r--    1 root     root           0 Jun 17 10:52 00000887
-rw-r--r--    1 root     root           0 Jun 17 10:59 00000888
-rw-r--r--    1 root     root           0 Jun 17 11:05 00000889
-rw-r--r--    1 root     root           0 Jun 17 11:12 00000890
-rw-r--r--    1 root     root           0 Jun 17 11:18 00000891
-rw-r--r--    1 root     root           0 Jun 17 11:25 00000892
-rw-r--r--    1 root     root           0 Jun 17 11:31 00000893
-rw-r--r--    1 root     root           0 Jun 17 11:38 00000894
-rw-r--r--    1 root     root           0 Jun 17 11:44 00000895
-rw-r--r--    1 root     root           0 Jun 17 13:01 00000896
-rw-r--r--    1 root     root           0 Jun 17 13:03 00000897
-rw-r--r--    1 root     root           0 Jun 17 13:05 00000898
-rw-r--r--    1 root     root           0 Jun 17 13:07 00000899
-rw-r--r--    1 root     root           0 Jun 17 13:09 00000900
-rw-r--r--    1 root     root           0 Jun 17 13:12 00000901
-rw-r--r--    1 root     root           0 Jun 17 15:01 00000902
-rw-r--r--    1 root     root           0 Jun 17 15:03 00000903
-rw-r--r--    1 root     root           0 Jun 17 15:04 00000904
-rw-r--r--    1 root     root           0 Jun 17 15:06 00000905
-rw-r--r--    1 root     root           0 Jun 17 15:09 00000906
-rw-r--r--    1 root     root           0 Jun 17 17:01 00000907
-rw-r--r--    1 root     root           0 Jun 17 17:03 00000908
-rw-r--r--    1 root     root           0 Jun 17 17:04 00000909
-rw-r--r--    1 root     root           0 Jun 17 17:06 00000910
-rw-r--r--    1 root     root           0 Jun 17 17:08 00000911
-rw-r--r--    1 root     root           0 Jun 17 19:04 00000912
-rw-r--r--    1 root     root           0 Jun 17 19:05 00000913
-rw-r--r--    1 root     root           0 Jun 17 19:07 00000914
-rw-r--r--    1 root     root           0 Jun 17 19:09 00000915
-rw-r--r--    1 root     root           0 Jun 17 19:11 00000916
-rw-r--r--    1 root     root           0 Jun 17 19:14 00000917
-rw-r--r--    1 root     root           0 Jun 17 21:04 00000918
-rw-r--r--    1 root     root           0 Jun 17 21:06 00000919
-rw-r--r--    1 root     root           0 Jun 17 21:07 00000920
-rw-r--r--    1 root     root           0 Jun 17 21:09 00000921
-rw-r--r--    1 root     root           0 Jun 17 21:12 00000922
-rw-r--r--    1 root     root           0 Jun 17 23:04 00000923
-rw-r--r--    1 root     root           0 Jun 17 23:05 00000924
-rw-r--r--    1 root     root           0 Jun 17 23:07 00000925
-rw-r--r--    1 root     root           0 Jun 17 23:09 00000926
-rw-r--r--    1 root     root           0 Jun 17 23:12 00000927
-rw-r--r--    1 root     root           0 Jun 17 23:15 00000928
-rw-r--r--    1 root     root           0 Jun 17 23:19 00000929
-rw-r--r--    1 root     root           0 Jun 18 01:04 00000930
-rw-r--r--    1 root     root           0 Jun 18 01:06 00000931
-rw-r--r--    1 root     root           0 Jun 18 01:08 00000932
-rw-r--r--    1 root     root           0 Jun 18 01:09 00000933
-rw-r--r--    1 root     root           0 Jun 18 01:12 00000934
-rw-r--r--    1 root     root           0 Jun 18 03:04 00000935
-rw-r--r--    1 root     root           0 Jun 18 03:06 00000936
-rw-r--r--    1 root     root           0 Jun 18 03:07 00000937
-rw-r--r--    1 root     root           0 Jun 18 03:09 00000938
-rw-r--r--    1 root     root           0 Jun 18 03:12 00000939
-rw-r--r--    1 root     root           0 Jun 18 03:15 00000940
-rw-r--r--    1 root     root           0 Jun 18 05:04 00000941
-rw-r--r--    1 root     root           0 Jun 18 05:05 00000942
-rw-r--r--    1 root     root           0 Jun 18 05:07 00000943
-rw-r--r--    1 root     root           0 Jun 18 05:09 00000944
-rw-r--r--    1 root     root           0 Jun 18 05:11 00000945
-rw-r--r--    1 root     root           0 Jun 18 05:14 00000946
-rw-r--r--    1 root     root           0 Jun 18 07:05 00000947
-rw-r--r--    1 root     root           0 Jun 18 07:06 00000948
-rw-r--r--    1 root     root           0 Jun 18 07:08 00000949
-rw-r--r--    1 root     root           0 Jun 18 07:09 00000950
-rw-r--r--    1 root     root           0 Jun 18 07:11 00000951
-rw-r--r--    1 root     root           0 Jun 18 07:14 00000952
-rw-r--r--    1 root     root           0 Jun 18 08:54 00000953
-rw-r--r--    1 root     root           0 Jun 18 08:56 00000954
-rw-r--r--    1 root     root           0 Jun 18 08:58 00000955
-rw-r--r--    1 root     root           0 Jun 18 09:00 00000956
-rw-r--r--    1 root     root           0 Jun 18 09:02 00000957
-rw-r--r--    1 root     root           0 Jun 18 09:05 00000958
drwxr-xr-x    2 root     root        4.0K May 13 13:09 checkpoint.00000034.tmp
drwxr-xr-x    2 root     root        4.0K May 18 03:16 checkpoint.00000168.tmp
drwxr-xr-x    2 root     root        4.0K May 19 19:19 checkpoint.00000237.tmp
drwxr-xr-x    2 root     root        4.0K May 27 09:07 checkpoint.00000465.tmp
drwxr-xr-x    2 root     root        4.0K May 28 13:22 checkpoint.00000481.tmp
drwxr-xr-x    2 root     root        4.0K May 31 09:11 checkpoint.00000488.tmp
drwxr-xr-x    2 root     root        4.0K May 31 13:12 checkpoint.00000492.tmp
drwxr-xr-x    2 root     root        4.0K May 31 15:16 checkpoint.00000493.tmp
drwxr-xr-x    2 root     root        4.0K Jun  1 11:10 checkpoint.00000516.tmp
drwxr-xr-x    2 root     root        4.0K Jun  1 13:07 checkpoint.00000517.tmp
drwxr-xr-x    2 root     root        4.0K Jun  1 19:07 checkpoint.00000525.tmp
drwxr-xr-x    2 root     root        4.0K Jun  1 21:12 checkpoint.00000528.tmp
drwxr-xr-x    2 root     root        4.0K Jun  2 01:07 checkpoint.00000532.tmp
drwxr-xr-x    2 root     root        4.0K Jun  2 03:06 checkpoint.00000533.tmp
drwxr-xr-x    2 root     root        4.0K Jun  2 09:07 checkpoint.00000540.tmp
drwxr-xr-x    2 root     root        4.0K Jun  2 15:15 checkpoint.00000547.tmp
drwxr-xr-x    2 root     root        4.0K Jun  2 17:17 checkpoint.00000548.tmp
drwxr-xr-x    2 root     root        4.0K Jun  2 19:08 checkpoint.00000549.tmp
drwxr-xr-x    2 root     root        4.0K Jun  2 21:07 checkpoint.00000551.tmp
drwxr-xr-x    2 root     root        4.0K Jun  2 23:07 checkpoint.00000552.tmp
drwxr-xr-x    2 root     root        4.0K Jun  3 03:06 checkpoint.00000554.tmp
drwxr-xr-x    2 root     root        4.0K Jun  3 05:07 checkpoint.00000555.tmp
drwxr-xr-x    2 root     root        4.0K Jun  3 07:07 checkpoint.00000557.tmp
drwxr-xr-x    2 root     root        4.0K Jun  3 09:07 checkpoint.00000558.tmp
drwxr-xr-x    2 root     root        4.0K Jun  3 17:15 checkpoint.00000573.tmp
drwxr-xr-x    2 root     root        4.0K Jun  3 19:20 checkpoint.00000574.tmp
drwxr-xr-x    2 root     root        4.0K Jun  3 21:07 checkpoint.00000576.tmp
drwxr-xr-x    2 root     root        4.0K Jun  3 23:07 checkpoint.00000577.tmp
drwxr-xr-x    2 root     root        4.0K Jun  4 01:07 checkpoint.00000578.tmp
drwxr-xr-x    2 root     root        4.0K Jun  4 03:06 checkpoint.00000580.tmp
drwxr-xr-x    2 root     root        4.0K Jun  4 05:06 checkpoint.00000581.tmp
drwxr-xr-x    2 root     root        4.0K Jun  4 07:07 checkpoint.00000582.tmp
drwxr-xr-x    2 root     root        4.0K Jun  4 09:17 checkpoint.00000584.tmp
drwxr-xr-x    2 root     root        4.0K Jun  4 11:18 checkpoint.00000590.tmp
drwxr-xr-x    2 root     root        4.0K Jun  4 13:18 checkpoint.00000592.tmp
drwxr-xr-x    2 root     root        4.0K Jun  4 15:13 checkpoint.00000593.tmp
drwxr-xr-x    2 root     root        4.0K Jun  4 17:07 checkpoint.00000594.tmp
drwxr-xr-x    2 root     root        4.0K Jun  4 21:07 checkpoint.00000596.tmp
drwxr-xr-x    2 root     root        4.0K Jun  4 23:07 checkpoint.00000597.tmp
drwxr-xr-x    2 root     root        4.0K Jun  5 01:06 checkpoint.00000598.tmp
drwxr-xr-x    2 root     root        4.0K Jun  5 03:07 checkpoint.00000600.tmp
drwxr-xr-x    2 root     root        4.0K Jun  5 05:06 checkpoint.00000601.tmp
drwxr-xr-x    2 root     root        4.0K Jun  5 07:06 checkpoint.00000602.tmp
drwxr-xr-x    2 root     root        4.0K Jun  5 09:06 checkpoint.00000604.tmp
drwxr-xr-x    2 root     root        4.0K Jun  5 11:06 checkpoint.00000605.tmp
drwxr-xr-x    2 root     root        4.0K Jun  5 15:06 checkpoint.00000607.tmp
drwxr-xr-x    2 root     root        4.0K Jun  6 11:06 checkpoint.00000618.tmp
drwxr-xr-x    2 root     root        4.0K Jun  6 13:06 checkpoint.00000619.tmp
drwxr-xr-x    2 root     root        4.0K Jun  6 17:06 checkpoint.00000621.tmp
drwxr-xr-x    2 root     root        4.0K Jun  6 21:06 checkpoint.00000622.tmp
drwxr-xr-x    2 root     root        4.0K Jun  7 03:06 checkpoint.00000624.tmp
drwxr-xr-x    2 root     root        4.0K Jun  7 05:06 checkpoint.00000625.tmp
drwxr-xr-x    2 root     root        4.0K Jun  7 07:06 checkpoint.00000627.tmp
drwxr-xr-x    2 root     root        4.0K Jun  7 09:13 checkpoint.00000639.tmp
drwxr-xr-x    2 root     root        4.0K Jun  7 17:08 checkpoint.00000641.tmp
drwxr-xr-x    2 root     root        4.0K Jun  8 11:19 checkpoint.00000650.tmp
drwxr-xr-x    2 root     root        4.0K Jun  8 13:18 checkpoint.00000651.tmp
drwxr-xr-x    2 root     root        4.0K Jun  8 15:17 checkpoint.00000653.tmp
drwxr-xr-x    2 root     root        4.0K Jun  8 17:21 checkpoint.00000654.tmp
drwxr-xr-x    2 root     root        4.0K Jun  8 19:21 checkpoint.00000655.tmp
drwxr-xr-x    2 root     root        4.0K Jun  8 21:19 checkpoint.00000657.tmp
drwxr-xr-x    2 root     root        4.0K Jun  8 23:16 checkpoint.00000658.tmp
drwxr-xr-x    2 root     root        4.0K Jun  9 03:13 checkpoint.00000659.tmp
drwxr-xr-x    2 root     root        4.0K Jun  9 05:13 checkpoint.00000661.tmp
drwxr-xr-x    2 root     root        4.0K Jun  9 07:14 checkpoint.00000668.tmp
drwxr-xr-x    2 root     root        4.0K Jun  9 09:24 checkpoint.00000669.tmp
drwxr-xr-x    2 root     root        4.0K Jun  9 11:19 checkpoint.00000671.tmp
drwxr-xr-x    2 root     root        4.0K Jun  9 13:26 checkpoint.00000672.tmp
drwxr-xr-x    2 root     root        4.0K Jun  9 15:20 checkpoint.00000673.tmp
drwxr-xr-x    2 root     root        4.0K Jun  9 15:29 checkpoint.00000675.tmp
drwxr-xr-x    2 root     root        4.0K Jun  9 17:24 checkpoint.00000676.tmp
drwxr-xr-x    2 root     root        4.0K Jun  9 19:25 checkpoint.00000677.tmp
drwxr-xr-x    2 root     root        4.0K Jun  9 21:22 checkpoint.00000679.tmp
drwxr-xr-x    2 root     root        4.0K Jun  9 23:21 checkpoint.00000680.tmp
drwxr-xr-x    2 root     root        4.0K Jun 10 01:21 checkpoint.00000681.tmp
drwxr-xr-x    2 root     root        4.0K Jun 10 03:26 checkpoint.00000690.tmp
drwxr-xr-x    2 root     root        4.0K Jun 10 05:26 checkpoint.00000692.tmp
drwxr-xr-x    2 root     root        4.0K Jun 10 07:35 checkpoint.00000693.tmp
drwxr-xr-x    2 root     root        4.0K Jun 10 09:25 checkpoint.00000694.tmp
drwxr-xr-x    2 root     root        4.0K Jun 10 11:27 checkpoint.00000696.tmp
drwxr-xr-x    2 root     root        4.0K Jun 10 12:38 checkpoint.00000697.tmp
drwxr-xr-x    2 root     root        4.0K Jun 10 15:20 checkpoint.00000700.tmp
drwxr-xr-x    2 root     root        4.0K Jun 14 11:07 checkpoint.00000701.tmp
drwxr-xr-x    2 root     root        4.0K Jun 14 13:15 checkpoint.00000709.tmp
drwxr-xr-x    2 root     root        4.0K Jun 14 14:08 checkpoint.00000711.tmp
drwxr-xr-x    2 root     root        4.0K Jun 14 15:12 checkpoint.00000712.tmp
drwxr-xr-x    2 root     root        4.0K Jun 14 17:17 checkpoint.00000713.tmp
drwxr-xr-x    2 root     root        4.0K Jun 14 19:08 checkpoint.00000715.tmp
drwxr-xr-x    2 root     root        4.0K Jun 14 21:12 checkpoint.00000721.tmp
drwxr-xr-x    2 root     root        4.0K Jun 14 23:07 checkpoint.00000723.tmp
drwxr-xr-x    2 root     root        4.0K Jun 15 01:08 checkpoint.00000724.tmp
drwxr-xr-x    2 root     root        4.0K Jun 15 03:07 checkpoint.00000725.tmp
drwxr-xr-x    2 root     root        4.0K Jun 15 05:07 checkpoint.00000727.tmp
drwxr-xr-x    2 root     root        4.0K Jun 15 07:07 checkpoint.00000728.tmp
drwxr-xr-x    2 root     root        4.0K Jun 15 09:12 checkpoint.00000734.tmp
drwxr-xr-x    2 root     root        4.0K Jun 15 11:16 checkpoint.00000736.tmp
drwxr-xr-x    2 root     root        4.0K Jun 15 13:12 checkpoint.00000737.tmp
drwxr-xr-x    2 root     root        4.0K Jun 15 15:15 checkpoint.00000738
drwxr-xr-x    2 root     root        4.0K Jun 15 15:20 checkpoint.00000742.tmp
drwxr-xr-x    2 root     root        4.0K Jun 15 16:12 checkpoint.00000743.tmp
drwxr-xr-x    2 root     root        4.0K Jun 15 17:26 checkpoint.00000745.tmp
drwxr-xr-x    2 root     root        4.0K Jun 15 19:20 checkpoint.00000747.tmp

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Comments: 17 (4 by maintainers)

Most upvoted comments

still a problem unsolved

As far as I can see, this issue is still present and waiting for feedback from the maintainers.

@bwplotka

We had the same issue today again. Receiver did not flush expired WAL files from disk until the disk was full. It had retention set to 6h yet kept WALs of the last 5 days around.

The way I still see it this issue might be fixed by the information I left in my previous comment. I be happy to read your thoughts on that.

Thanks!