thanos: receive: does not cleanup old data on disk, but tries to load it on boot, causing OOM down the line
Thanos, Prometheus and Golang version used: quay.io/thanos/thanos:v0.21.1
Object Storage Provider: GCS
What happened: Running with:
--tsdb.retention=1h
--tsdb.path=/thanos-data
--tsdb.allow-overlapping-blocks
--shipper.allow-out-of-order-uploads
Several days old data is left on disk, with no attempts of shipping it to storage. Not necessarily days worth of data, tho I have not really checked that.
Some WAL segments are left forever on disk, eventually causing OOM on boot, as memory needs becomes ever growing. Giving it enough memory to load all segments appears to make no difference, as even when loaded no attempts to ship them are made, even after several days.
Manual removal of the affected segments on disk appears to resolve it, as nothing is logged about data inconsistency after doing so.
What you expected to happen:
Old data is removed from disk and not attempted to be loaded forever.
How to reproduce it (as minimally and precisely as possible): Run a receiver and ingest data to it, observe data being left on disk. It may need to run for a few days, and restart a few times, maybe be shutdown for sometime in-between too, not entirely sure.
Full logs to relevant components:
Looking in the WAL directory (captured a few weeks ago when first encountering it)
/ # ls -lah /thanos-data/default-tenant/wal/
total 127M
drwxr-xr-x 104 root root 12.0K Jun 18 09:05 .
drwxr-xr-x 6 root root 4.0K Jun 18 09:05 ..
-rw-r--r-- 1 root root 3.2M Jun 15 09:12 00000739
-rw-r--r-- 1 root root 96.0K Jun 15 11:16 00000740
-rw-r--r-- 1 root root 10.5K Jun 15 11:16 00000741
-rw-r--r-- 1 root root 0 Jun 15 13:03 00000742
-rw-r--r-- 1 root root 0 Jun 15 13:11 00000743
-rw-r--r-- 1 root root 64.0K Jun 15 15:15 00000744
-rw-r--r-- 1 root root 15.3M Jun 15 15:20 00000745
-rw-r--r-- 1 root root 4.6M Jun 15 15:20 00000746
-rw-r--r-- 1 root root 29.4M Jun 15 16:12 00000747
-rw-r--r-- 1 root root 7.6M Jun 15 16:13 00000748
-rw-r--r-- 1 root root 0 Jun 15 17:01 00000749
-rw-r--r-- 1 root root 15.3M Jun 15 17:26 00000750
-rw-r--r-- 1 root root 0 Jun 15 17:26 00000751
-rw-r--r-- 1 root root 45.3M Jun 15 19:20 00000752
-rw-r--r-- 1 root root 5.6M Jun 15 19:20 00000753
-rw-r--r-- 1 root root 0 Jun 15 21:05 00000754
-rw-r--r-- 1 root root 0 Jun 15 21:07 00000755
-rw-r--r-- 1 root root 0 Jun 15 23:06 00000756
-rw-r--r-- 1 root root 0 Jun 16 01:05 00000757
-rw-r--r-- 1 root root 0 Jun 16 01:07 00000758
-rw-r--r-- 1 root root 0 Jun 16 03:06 00000759
-rw-r--r-- 1 root root 0 Jun 16 03:07 00000760
-rw-r--r-- 1 root root 0 Jun 16 05:05 00000761
-rw-r--r-- 1 root root 0 Jun 16 06:49 00000762
-rw-r--r-- 1 root root 0 Jun 16 06:50 00000763
-rw-r--r-- 1 root root 0 Jun 16 06:52 00000764
-rw-r--r-- 1 root root 0 Jun 16 07:03 00000765
-rw-r--r-- 1 root root 0 Jun 16 07:05 00000766
-rw-r--r-- 1 root root 0 Jun 16 07:07 00000767
-rw-r--r-- 1 root root 0 Jun 16 09:03 00000768
-rw-r--r-- 1 root root 0 Jun 16 09:04 00000769
-rw-r--r-- 1 root root 0 Jun 16 09:06 00000770
-rw-r--r-- 1 root root 0 Jun 16 09:08 00000771
-rw-r--r-- 1 root root 0 Jun 16 09:10 00000772
-rw-r--r-- 1 root root 0 Jun 16 09:13 00000773
-rw-r--r-- 1 root root 0 Jun 16 09:17 00000774
-rw-r--r-- 1 root root 0 Jun 16 09:24 00000775
-rw-r--r-- 1 root root 0 Jun 16 10:23 00000776
-rw-r--r-- 1 root root 0 Jun 16 10:25 00000777
-rw-r--r-- 1 root root 0 Jun 16 10:27 00000778
-rw-r--r-- 1 root root 0 Jun 16 10:28 00000779
-rw-r--r-- 1 root root 0 Jun 16 10:31 00000780
-rw-r--r-- 1 root root 0 Jun 16 10:34 00000781
-rw-r--r-- 1 root root 0 Jun 16 10:38 00000782
-rw-r--r-- 1 root root 0 Jun 16 10:44 00000783
-rw-r--r-- 1 root root 0 Jun 16 10:51 00000784
-rw-r--r-- 1 root root 0 Jun 16 10:57 00000785
-rw-r--r-- 1 root root 0 Jun 16 11:04 00000786
-rw-r--r-- 1 root root 0 Jun 16 11:11 00000787
-rw-r--r-- 1 root root 0 Jun 16 11:17 00000788
-rw-r--r-- 1 root root 0 Jun 16 11:24 00000789
-rw-r--r-- 1 root root 0 Jun 16 11:30 00000790
-rw-r--r-- 1 root root 0 Jun 16 11:37 00000791
-rw-r--r-- 1 root root 0 Jun 16 11:54 00000792
-rw-r--r-- 1 root root 0 Jun 16 11:55 00000793
-rw-r--r-- 1 root root 0 Jun 16 11:57 00000794
-rw-r--r-- 1 root root 0 Jun 16 11:59 00000795
-rw-r--r-- 1 root root 0 Jun 16 12:01 00000796
-rw-r--r-- 1 root root 0 Jun 16 12:03 00000797
-rw-r--r-- 1 root root 0 Jun 16 12:07 00000798
-rw-r--r-- 1 root root 0 Jun 16 12:14 00000799
-rw-r--r-- 1 root root 0 Jun 16 12:21 00000800
-rw-r--r-- 1 root root 0 Jun 16 12:27 00000801
-rw-r--r-- 1 root root 0 Jun 16 12:33 00000802
-rw-r--r-- 1 root root 0 Jun 16 12:34 00000803
-rw-r--r-- 1 root root 0 Jun 16 12:36 00000804
-rw-r--r-- 1 root root 0 Jun 16 12:38 00000805
-rw-r--r-- 1 root root 0 Jun 16 12:40 00000806
-rw-r--r-- 1 root root 0 Jun 16 12:43 00000807
-rw-r--r-- 1 root root 0 Jun 16 12:48 00000808
-rw-r--r-- 1 root root 0 Jun 16 12:54 00000809
-rw-r--r-- 1 root root 0 Jun 16 13:01 00000810
-rw-r--r-- 1 root root 0 Jun 16 13:07 00000811
-rw-r--r-- 1 root root 0 Jun 16 13:14 00000812
-rw-r--r-- 1 root root 0 Jun 16 13:20 00000813
-rw-r--r-- 1 root root 0 Jun 16 13:27 00000814
-rw-r--r-- 1 root root 0 Jun 16 14:38 00000815
-rw-r--r-- 1 root root 0 Jun 16 14:39 00000816
-rw-r--r-- 1 root root 0 Jun 16 14:41 00000817
-rw-r--r-- 1 root root 0 Jun 16 14:43 00000818
-rw-r--r-- 1 root root 0 Jun 16 14:45 00000819
-rw-r--r-- 1 root root 0 Jun 16 14:48 00000820
-rw-r--r-- 1 root root 0 Jun 16 14:52 00000821
-rw-r--r-- 1 root root 0 Jun 16 14:59 00000822
-rw-r--r-- 1 root root 0 Jun 16 15:05 00000823
-rw-r--r-- 1 root root 0 Jun 16 15:12 00000824
-rw-r--r-- 1 root root 0 Jun 16 15:18 00000825
-rw-r--r-- 1 root root 0 Jun 16 16:40 00000826
-rw-r--r-- 1 root root 0 Jun 16 16:42 00000827
-rw-r--r-- 1 root root 0 Jun 16 16:43 00000828
-rw-r--r-- 1 root root 0 Jun 16 16:45 00000829
-rw-r--r-- 1 root root 0 Jun 16 16:47 00000830
-rw-r--r-- 1 root root 0 Jun 16 16:50 00000831
-rw-r--r-- 1 root root 0 Jun 16 16:54 00000832
-rw-r--r-- 1 root root 0 Jun 16 17:01 00000833
-rw-r--r-- 1 root root 0 Jun 16 17:07 00000834
-rw-r--r-- 1 root root 0 Jun 16 19:04 00000835
-rw-r--r-- 1 root root 0 Jun 16 19:05 00000836
-rw-r--r-- 1 root root 0 Jun 16 19:07 00000837
-rw-r--r-- 1 root root 0 Jun 16 19:09 00000838
-rw-r--r-- 1 root root 0 Jun 16 19:11 00000839
-rw-r--r-- 1 root root 0 Jun 16 19:14 00000840
-rw-r--r-- 1 root root 0 Jun 16 19:18 00000841
-rw-r--r-- 1 root root 0 Jun 16 19:24 00000842
-rw-r--r-- 1 root root 0 Jun 16 21:02 00000843
-rw-r--r-- 1 root root 0 Jun 16 21:04 00000844
-rw-r--r-- 1 root root 0 Jun 16 21:06 00000845
-rw-r--r-- 1 root root 0 Jun 16 23:03 00000846
-rw-r--r-- 1 root root 0 Jun 16 23:05 00000847
-rw-r--r-- 1 root root 0 Jun 16 23:07 00000848
-rw-r--r-- 1 root root 0 Jun 16 23:09 00000849
-rw-r--r-- 1 root root 0 Jun 17 01:06 00000850
-rw-r--r-- 1 root root 0 Jun 17 01:07 00000851
-rw-r--r-- 1 root root 0 Jun 17 03:05 00000852
-rw-r--r-- 1 root root 0 Jun 17 03:07 00000853
-rw-r--r-- 1 root root 0 Jun 17 03:08 00000854
-rw-r--r-- 1 root root 0 Jun 17 05:06 00000855
-rw-r--r-- 1 root root 0 Jun 17 05:07 00000856
-rw-r--r-- 1 root root 0 Jun 17 07:01 00000857
-rw-r--r-- 1 root root 0 Jun 17 07:03 00000858
-rw-r--r-- 1 root root 0 Jun 17 07:05 00000859
-rw-r--r-- 1 root root 0 Jun 17 07:08 00000860
-rw-r--r-- 1 root root 0 Jun 17 07:10 00000861
-rw-r--r-- 1 root root 0 Jun 17 07:26 00000862
-rw-r--r-- 1 root root 0 Jun 17 07:28 00000863
-rw-r--r-- 1 root root 0 Jun 17 07:29 00000864
-rw-r--r-- 1 root root 0 Jun 17 07:31 00000865
-rw-r--r-- 1 root root 0 Jun 17 09:01 00000866
-rw-r--r-- 1 root root 0 Jun 17 09:03 00000867
-rw-r--r-- 1 root root 0 Jun 17 09:04 00000868
-rw-r--r-- 1 root root 0 Jun 17 09:06 00000869
-rw-r--r-- 1 root root 0 Jun 17 09:08 00000870
-rw-r--r-- 1 root root 0 Jun 17 09:11 00000871
-rw-r--r-- 1 root root 0 Jun 17 09:15 00000872
-rw-r--r-- 1 root root 0 Jun 17 09:21 00000873
-rw-r--r-- 1 root root 0 Jun 17 09:28 00000874
-rw-r--r-- 1 root root 0 Jun 17 09:34 00000875
-rw-r--r-- 1 root root 0 Jun 17 09:41 00000876
-rw-r--r-- 1 root root 0 Jun 17 09:47 00000877
-rw-r--r-- 1 root root 0 Jun 17 09:53 00000878
-rw-r--r-- 1 root root 0 Jun 17 10:00 00000879
-rw-r--r-- 1 root root 0 Jun 17 10:06 00000880
-rw-r--r-- 1 root root 0 Jun 17 10:13 00000881
-rw-r--r-- 1 root root 0 Jun 17 10:19 00000882
-rw-r--r-- 1 root root 0 Jun 17 10:26 00000883
-rw-r--r-- 1 root root 0 Jun 17 10:32 00000884
-rw-r--r-- 1 root root 0 Jun 17 10:39 00000885
-rw-r--r-- 1 root root 0 Jun 17 10:45 00000886
-rw-r--r-- 1 root root 0 Jun 17 10:52 00000887
-rw-r--r-- 1 root root 0 Jun 17 10:59 00000888
-rw-r--r-- 1 root root 0 Jun 17 11:05 00000889
-rw-r--r-- 1 root root 0 Jun 17 11:12 00000890
-rw-r--r-- 1 root root 0 Jun 17 11:18 00000891
-rw-r--r-- 1 root root 0 Jun 17 11:25 00000892
-rw-r--r-- 1 root root 0 Jun 17 11:31 00000893
-rw-r--r-- 1 root root 0 Jun 17 11:38 00000894
-rw-r--r-- 1 root root 0 Jun 17 11:44 00000895
-rw-r--r-- 1 root root 0 Jun 17 13:01 00000896
-rw-r--r-- 1 root root 0 Jun 17 13:03 00000897
-rw-r--r-- 1 root root 0 Jun 17 13:05 00000898
-rw-r--r-- 1 root root 0 Jun 17 13:07 00000899
-rw-r--r-- 1 root root 0 Jun 17 13:09 00000900
-rw-r--r-- 1 root root 0 Jun 17 13:12 00000901
-rw-r--r-- 1 root root 0 Jun 17 15:01 00000902
-rw-r--r-- 1 root root 0 Jun 17 15:03 00000903
-rw-r--r-- 1 root root 0 Jun 17 15:04 00000904
-rw-r--r-- 1 root root 0 Jun 17 15:06 00000905
-rw-r--r-- 1 root root 0 Jun 17 15:09 00000906
-rw-r--r-- 1 root root 0 Jun 17 17:01 00000907
-rw-r--r-- 1 root root 0 Jun 17 17:03 00000908
-rw-r--r-- 1 root root 0 Jun 17 17:04 00000909
-rw-r--r-- 1 root root 0 Jun 17 17:06 00000910
-rw-r--r-- 1 root root 0 Jun 17 17:08 00000911
-rw-r--r-- 1 root root 0 Jun 17 19:04 00000912
-rw-r--r-- 1 root root 0 Jun 17 19:05 00000913
-rw-r--r-- 1 root root 0 Jun 17 19:07 00000914
-rw-r--r-- 1 root root 0 Jun 17 19:09 00000915
-rw-r--r-- 1 root root 0 Jun 17 19:11 00000916
-rw-r--r-- 1 root root 0 Jun 17 19:14 00000917
-rw-r--r-- 1 root root 0 Jun 17 21:04 00000918
-rw-r--r-- 1 root root 0 Jun 17 21:06 00000919
-rw-r--r-- 1 root root 0 Jun 17 21:07 00000920
-rw-r--r-- 1 root root 0 Jun 17 21:09 00000921
-rw-r--r-- 1 root root 0 Jun 17 21:12 00000922
-rw-r--r-- 1 root root 0 Jun 17 23:04 00000923
-rw-r--r-- 1 root root 0 Jun 17 23:05 00000924
-rw-r--r-- 1 root root 0 Jun 17 23:07 00000925
-rw-r--r-- 1 root root 0 Jun 17 23:09 00000926
-rw-r--r-- 1 root root 0 Jun 17 23:12 00000927
-rw-r--r-- 1 root root 0 Jun 17 23:15 00000928
-rw-r--r-- 1 root root 0 Jun 17 23:19 00000929
-rw-r--r-- 1 root root 0 Jun 18 01:04 00000930
-rw-r--r-- 1 root root 0 Jun 18 01:06 00000931
-rw-r--r-- 1 root root 0 Jun 18 01:08 00000932
-rw-r--r-- 1 root root 0 Jun 18 01:09 00000933
-rw-r--r-- 1 root root 0 Jun 18 01:12 00000934
-rw-r--r-- 1 root root 0 Jun 18 03:04 00000935
-rw-r--r-- 1 root root 0 Jun 18 03:06 00000936
-rw-r--r-- 1 root root 0 Jun 18 03:07 00000937
-rw-r--r-- 1 root root 0 Jun 18 03:09 00000938
-rw-r--r-- 1 root root 0 Jun 18 03:12 00000939
-rw-r--r-- 1 root root 0 Jun 18 03:15 00000940
-rw-r--r-- 1 root root 0 Jun 18 05:04 00000941
-rw-r--r-- 1 root root 0 Jun 18 05:05 00000942
-rw-r--r-- 1 root root 0 Jun 18 05:07 00000943
-rw-r--r-- 1 root root 0 Jun 18 05:09 00000944
-rw-r--r-- 1 root root 0 Jun 18 05:11 00000945
-rw-r--r-- 1 root root 0 Jun 18 05:14 00000946
-rw-r--r-- 1 root root 0 Jun 18 07:05 00000947
-rw-r--r-- 1 root root 0 Jun 18 07:06 00000948
-rw-r--r-- 1 root root 0 Jun 18 07:08 00000949
-rw-r--r-- 1 root root 0 Jun 18 07:09 00000950
-rw-r--r-- 1 root root 0 Jun 18 07:11 00000951
-rw-r--r-- 1 root root 0 Jun 18 07:14 00000952
-rw-r--r-- 1 root root 0 Jun 18 08:54 00000953
-rw-r--r-- 1 root root 0 Jun 18 08:56 00000954
-rw-r--r-- 1 root root 0 Jun 18 08:58 00000955
-rw-r--r-- 1 root root 0 Jun 18 09:00 00000956
-rw-r--r-- 1 root root 0 Jun 18 09:02 00000957
-rw-r--r-- 1 root root 0 Jun 18 09:05 00000958
drwxr-xr-x 2 root root 4.0K May 13 13:09 checkpoint.00000034.tmp
drwxr-xr-x 2 root root 4.0K May 18 03:16 checkpoint.00000168.tmp
drwxr-xr-x 2 root root 4.0K May 19 19:19 checkpoint.00000237.tmp
drwxr-xr-x 2 root root 4.0K May 27 09:07 checkpoint.00000465.tmp
drwxr-xr-x 2 root root 4.0K May 28 13:22 checkpoint.00000481.tmp
drwxr-xr-x 2 root root 4.0K May 31 09:11 checkpoint.00000488.tmp
drwxr-xr-x 2 root root 4.0K May 31 13:12 checkpoint.00000492.tmp
drwxr-xr-x 2 root root 4.0K May 31 15:16 checkpoint.00000493.tmp
drwxr-xr-x 2 root root 4.0K Jun 1 11:10 checkpoint.00000516.tmp
drwxr-xr-x 2 root root 4.0K Jun 1 13:07 checkpoint.00000517.tmp
drwxr-xr-x 2 root root 4.0K Jun 1 19:07 checkpoint.00000525.tmp
drwxr-xr-x 2 root root 4.0K Jun 1 21:12 checkpoint.00000528.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 01:07 checkpoint.00000532.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 03:06 checkpoint.00000533.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 09:07 checkpoint.00000540.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 15:15 checkpoint.00000547.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 17:17 checkpoint.00000548.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 19:08 checkpoint.00000549.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 21:07 checkpoint.00000551.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 23:07 checkpoint.00000552.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 03:06 checkpoint.00000554.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 05:07 checkpoint.00000555.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 07:07 checkpoint.00000557.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 09:07 checkpoint.00000558.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 17:15 checkpoint.00000573.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 19:20 checkpoint.00000574.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 21:07 checkpoint.00000576.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 23:07 checkpoint.00000577.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 01:07 checkpoint.00000578.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 03:06 checkpoint.00000580.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 05:06 checkpoint.00000581.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 07:07 checkpoint.00000582.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 09:17 checkpoint.00000584.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 11:18 checkpoint.00000590.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 13:18 checkpoint.00000592.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 15:13 checkpoint.00000593.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 17:07 checkpoint.00000594.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 21:07 checkpoint.00000596.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 23:07 checkpoint.00000597.tmp
drwxr-xr-x 2 root root 4.0K Jun 5 01:06 checkpoint.00000598.tmp
drwxr-xr-x 2 root root 4.0K Jun 5 03:07 checkpoint.00000600.tmp
drwxr-xr-x 2 root root 4.0K Jun 5 05:06 checkpoint.00000601.tmp
drwxr-xr-x 2 root root 4.0K Jun 5 07:06 checkpoint.00000602.tmp
drwxr-xr-x 2 root root 4.0K Jun 5 09:06 checkpoint.00000604.tmp
drwxr-xr-x 2 root root 4.0K Jun 5 11:06 checkpoint.00000605.tmp
drwxr-xr-x 2 root root 4.0K Jun 5 15:06 checkpoint.00000607.tmp
drwxr-xr-x 2 root root 4.0K Jun 6 11:06 checkpoint.00000618.tmp
drwxr-xr-x 2 root root 4.0K Jun 6 13:06 checkpoint.00000619.tmp
drwxr-xr-x 2 root root 4.0K Jun 6 17:06 checkpoint.00000621.tmp
drwxr-xr-x 2 root root 4.0K Jun 6 21:06 checkpoint.00000622.tmp
drwxr-xr-x 2 root root 4.0K Jun 7 03:06 checkpoint.00000624.tmp
drwxr-xr-x 2 root root 4.0K Jun 7 05:06 checkpoint.00000625.tmp
drwxr-xr-x 2 root root 4.0K Jun 7 07:06 checkpoint.00000627.tmp
drwxr-xr-x 2 root root 4.0K Jun 7 09:13 checkpoint.00000639.tmp
drwxr-xr-x 2 root root 4.0K Jun 7 17:08 checkpoint.00000641.tmp
drwxr-xr-x 2 root root 4.0K Jun 8 11:19 checkpoint.00000650.tmp
drwxr-xr-x 2 root root 4.0K Jun 8 13:18 checkpoint.00000651.tmp
drwxr-xr-x 2 root root 4.0K Jun 8 15:17 checkpoint.00000653.tmp
drwxr-xr-x 2 root root 4.0K Jun 8 17:21 checkpoint.00000654.tmp
drwxr-xr-x 2 root root 4.0K Jun 8 19:21 checkpoint.00000655.tmp
drwxr-xr-x 2 root root 4.0K Jun 8 21:19 checkpoint.00000657.tmp
drwxr-xr-x 2 root root 4.0K Jun 8 23:16 checkpoint.00000658.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 03:13 checkpoint.00000659.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 05:13 checkpoint.00000661.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 07:14 checkpoint.00000668.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 09:24 checkpoint.00000669.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 11:19 checkpoint.00000671.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 13:26 checkpoint.00000672.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 15:20 checkpoint.00000673.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 15:29 checkpoint.00000675.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 17:24 checkpoint.00000676.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 19:25 checkpoint.00000677.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 21:22 checkpoint.00000679.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 23:21 checkpoint.00000680.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 01:21 checkpoint.00000681.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 03:26 checkpoint.00000690.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 05:26 checkpoint.00000692.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 07:35 checkpoint.00000693.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 09:25 checkpoint.00000694.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 11:27 checkpoint.00000696.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 12:38 checkpoint.00000697.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 15:20 checkpoint.00000700.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 11:07 checkpoint.00000701.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 13:15 checkpoint.00000709.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 14:08 checkpoint.00000711.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 15:12 checkpoint.00000712.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 17:17 checkpoint.00000713.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 19:08 checkpoint.00000715.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 21:12 checkpoint.00000721.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 23:07 checkpoint.00000723.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 01:08 checkpoint.00000724.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 03:07 checkpoint.00000725.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 05:07 checkpoint.00000727.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 07:07 checkpoint.00000728.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 09:12 checkpoint.00000734.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 11:16 checkpoint.00000736.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 13:12 checkpoint.00000737.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 15:15 checkpoint.00000738
drwxr-xr-x 2 root root 4.0K Jun 15 15:20 checkpoint.00000742.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 16:12 checkpoint.00000743.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 17:26 checkpoint.00000745.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 19:20 checkpoint.00000747.tmp
About this issue
- Original URL
- State: open
- Created 5 years ago
- Comments: 17 (4 by maintainers)
still a problem unsolved
As far as I can see, this issue is still present and waiting for feedback from the maintainers.
@bwplotka
We had the same issue today again. Receiver did not flush expired WAL files from disk until the disk was full. It had retention set to 6h yet kept WALs of the last 5 days around.
The way I still see it this issue might be fixed by the information I left in my previous comment. I be happy to read your thoughts on that.
Thanks!