ClickHouse: Clickhouse do not stop generating files on s3

Describe what’s wrong

I was trying to test Clickhouse with s3 storage. I created test table and it created more than 35gb and keep continue to generate new files even though table creation was finished in 3 seconds with less than 1gb.

How to reproduce

I followed this post (you can reproduce). I am using is Cloudflare R2 as S3 storage.

Clickhouse is running on k3s and the version is clickhouse/clickhouse-server:23.3-alpine

Expected behavior

A clear and concise description of what you expected to happen.

Error message and/or stacktrace

2023.07.20 23:55:32.860850 [ 238 ] {bbc5517c-ba9b-4a69-9a68-9c2ff4b5c3c6::all_1_10_1} <Error> DiskObjectStorageTransaction: An error occurred while executing transaction's operation #0 (RemoveManyObjectStorageOperation (paths size: 13, keep all batch true, files to keep )): Code: 107. DB::Exception: Metadata path 'store/bbc/bbc5517c-ba9b-4a69-9a68-9c2ff4b5c3c6/delete_tmp_tmp_merge_all_1_10_1/msg.bin' doesn't exist. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0xe22f215 in /usr/bin/clickhouse
1. ? @ 0x8db6d24 in /usr/bin/clickhouse
2. ? @ 0x12cdd9f6 in /usr/bin/clickhouse
3. DB::DiskObjectStorageTransaction::commit() @ 0x12cd8f7d in /usr/bin/clickhouse
4. DB::DiskObjectStorage::removeSharedFiles(std::vector<DB::RemoveRequest, std::allocator<DB::RemoveRequest>> const&, bool, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&) @ 0x12cb9ea5 in /usr/bin/clickhouse
5. DB::DataPartStorageOnDiskBase::clearDirectory(String const&, DB::CanRemoveDescription const&, DB::MergeTreeDataPartChecksums const&, bool, Poco::Logger*) @ 0x14542381 in /usr/bin/clickhouse
6. DB::DataPartStorageOnDiskBase::remove(std::function<DB::CanRemoveDescription ()>&&, DB::MergeTreeDataPartChecksums const&, std::list<DB::IDataPartStorage::ProjectionChecksums, std::allocator<DB::IDataPartStorage::ProjectionChecksums>>, bool, Poco::Logger*) @ 0x1453ef20 in /usr/bin/clickhouse
7. DB::IMergeTreeDataPart::remove() @ 0x145575b7 in /usr/bin/clickhouse
8. DB::IMergeTreeDataPart::removeIfNeeded() @ 0x14556987 in /usr/bin/clickhouse
9. DB::MergeTreeDataPartWide::~MergeTreeDataPartWide() @ 0x146889d0 in /usr/bin/clickhouse
10. DB::IMergeTreeDataPartWriter::~IMergeTreeDataPartWriter() @ 0x14692797 in /usr/bin/clickhouse
11. ? @ 0x146a606c in /usr/bin/clickhouse
12. ? @ 0x145178d3 in /usr/bin/clickhouse
13. ? @ 0x145c16e4 in /usr/bin/clickhouse
14. ? @ 0x145c2610 in /usr/bin/clickhouse
15. ? @ 0x149499e9 in /usr/bin/clickhouse
16. DB::MergePlainMergeTreeTask::executeStep() @ 0x149487ef in /usr/bin/clickhouse
17. DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x8a1f08c in /usr/bin/clickhouse
18. DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x8a1ed0a in /usr/bin/clickhouse
19. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0xe302785 in /usr/bin/clickhouse
20. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0xe3052f5 in /usr/bin/clickhouse
21. ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0xe2fe553 in /usr/bin/clickhouse
22. ? @ 0xe3041c1 in /usr/bin/clickhouse
23. ? @ 0x7f5150a56609 in ?
24. __clone @ 0x7f515097b133 in ?
 (version 23.3.8.21 (official build))

2023.07.20 23:55:40.396393 [ 239 ] {bbc5517c-ba9b-4a69-9a68-9c2ff4b5c3c6::all_1_10_1} <Error> DiskObjectStorageTransaction: An error occurred while executing transaction's operation #0 (RemoveManyObjectStorageOperation (paths size: 13, keep all batch true, files to keep )): Code: 107. DB::Exception: Metadata path 'store/bbc/bbc5517c-ba9b-4a69-9a68-9c2ff4b5c3c6/delete_tmp_tmp_merge_all_1_10_1/msg.bin' doesn't exist. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0xe22f215 in /usr/bin/clickhouse
1. ? @ 0x8db6d24 in /usr/bin/clickhouse
2. ? @ 0x12cdd9f6 in /usr/bin/clickhouse
3. DB::DiskObjectStorageTransaction::commit() @ 0x12cd8f7d in /usr/bin/clickhouse
4. DB::DiskObjectStorage::removeSharedFiles(std::vector<DB::RemoveRequest, std::allocator<DB::RemoveRequest>> const&, bool, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&) @ 0x12cb9ea5 in /usr/bin/clickhouse
5. DB::DataPartStorageOnDiskBase::clearDirectory(String const&, DB::CanRemoveDescription const&, DB::MergeTreeDataPartChecksums const&, bool, Poco::Logger*) @ 0x14542381 in /usr/bin/clickhouse
6. DB::DataPartStorageOnDiskBase::remove(std::function<DB::CanRemoveDescription ()>&&, DB::MergeTreeDataPartChecksums const&, std::list<DB::IDataPartStorage::ProjectionChecksums, std::allocator<DB::IDataPartStorage::ProjectionChecksums>>, bool, Poco::Logger*) @ 0x1453ef20 in /usr/bin/clickhouse
7. DB::IMergeTreeDataPart::remove() @ 0x145575b7 in /usr/bin/clickhouse
8. DB::IMergeTreeDataPart::removeIfNeeded() @ 0x14556987 in /usr/bin/clickhouse
9. DB::MergeTreeDataPartWide::~MergeTreeDataPartWide() @ 0x146889d0 in /usr/bin/clickhouse
10. DB::IMergeTreeDataPartWriter::~IMergeTreeDataPartWriter() @ 0x14692797 in /usr/bin/clickhouse
11. ? @ 0x146a606c in /usr/bin/clickhouse
12. ? @ 0x145178d3 in /usr/bin/clickhouse
13. ? @ 0x145c16e4 in /usr/bin/clickhouse
14. ? @ 0x145c2610 in /usr/bin/clickhouse
15. ? @ 0x149499e9 in /usr/bin/clickhouse
16. DB::MergePlainMergeTreeTask::executeStep() @ 0x149487ef in /usr/bin/clickhouse
17. DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x8a1f08c in /usr/bin/clickhouse
18. DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x8a1ed0a in /usr/bin/clickhouse
19. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0xe302785 in /usr/bin/clickhouse
20. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0xe3052f5 in /usr/bin/clickhouse
21. ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0xe2fe553 in /usr/bin/clickhouse
22. ? @ 0xe3041c1 in /usr/bin/clickhouse
23. ? @ 0x7f5150a56609 in ?
24. __clone @ 0x7f515097b133 in ?
 (version 23.3.8.21 (official build))
2023.07.20 23:55:40.396535 [ 239 ] {bbc5517c-ba9b-4a69-9a68-9c2ff4b5c3c6::all_1_10_1} <Error> default.test (bbc5517c-ba9b-4a69-9a68-9c2ff4b5c3c6): Cannot quickly remove directory /var/lib/clickhouse/disks/s3/store/bbc/bbc5517c-ba9b-4a69-9a68-9c2ff4b5c3c6/delete_tmp_tmp_merge_all_1_10_1 by removing files; fallback to recursive removal. Reason: Code: 107. DB::Exception: Metadata path 'store/bbc/bbc5517c-ba9b-4a69-9a68-9c2ff4b5c3c6/delete_tmp_tmp_merge_all_1_10_1/msg.bin' doesn't exist. (FILE_DOESNT_EXIST) (version 23.3.8.21 (official build))
2023.07.20 23:55:40.397157 [ 239 ] {} <Error> void DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(DB::TaskRuntimeDataPtr) [Queue = DB::DynamicRuntimeQueue]: Code: 499. DB::Exception: Message: Unable to parse ExceptionName: InvalidPart Message: All non-trailing parts must have the same length., Key: db/lqc/dycgsmkvquvhttwolkldknmcwrgaz, Bucket: cgnr-clickhouse, Tags: "2f4c12d4724f6a6138697977d1125d1e" "7354c10c2c0074eb705597de9528bacf" "c2e58bee960d677fdcc3b8ef70b6af1d" "dee259a560e342de2242fbbaecc52262" "c3342ced8dea2fa90d3e207f67b00725" "cd1d1100a3e1605581c75331b5d9d13a". (S3_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0xe22f215 in /usr/bin/clickhouse
1. ? @ 0x1286b488 in /usr/bin/clickhouse
2. DB::WriteBufferFromS3::completeMultipartUpload() @ 0x1286a062 in /usr/bin/clickhouse
3. DB::WriteBufferFromS3::finalizeImpl() @ 0x128681d8 in /usr/bin/clickhouse
4. ? @ 0x8a5aac8 in /usr/bin/clickhouse
5. DB::WriteIndirectBufferFromRemoteFS::finalizeImpl() @ 0x12d1a269 in /usr/bin/clickhouse
6. ? @ 0x8a5aac8 in /usr/bin/clickhouse
7. DB::MergeTreeDataPartWriterWide::finishDataSerialization(bool) @ 0x146a59e7 in /usr/bin/clickhouse
8. DB::MergeTreeDataPartWriterWide::finish(bool) @ 0x146a5c3f in /usr/bin/clickhouse
9. DB::MergedBlockOutputStream::Finalizer::Impl::finish() @ 0x147d72f6 in /usr/bin/clickhouse
10. DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x147d7579 in /usr/bin/clickhouse
11. DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x145bcce9 in /usr/bin/clickhouse
12. bool std::__function::__policy_invoker<bool ()>::__call_impl<std::__function::__default_alloc_func<DB::MergeTask::MergeProjectionsStage::subtasks::'lambda1'(), bool ()>>(std::__function::__policy_storage const*) @ 0x145c0e89 in /usr/bin/clickhouse
13. DB::MergeTask::MergeProjectionsStage::execute() @ 0x145bcf4b in /usr/bin/clickhouse
14. DB::MergeTask::execute() @ 0x145bc91a in /usr/bin/clickhouse
15. DB::MergePlainMergeTreeTask::executeStep() @ 0x14948711 in /usr/bin/clickhouse
16. DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x8a1f08c in /usr/bin/clickhouse
17. DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x8a1ed0a in /usr/bin/clickhouse
18. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0xe302785 in /usr/bin/clickhouse
19. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0xe3052f5 in /usr/bin/clickhouse
20. ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0xe2fe553 in /usr/bin/clickhouse
21. ? @ 0xe3041c1 in /usr/bin/clickhouse
22. ? @ 0x7f5150a56609 in ?
23. __clone @ 0x7f515097b133 in ?
 (version 23.3.8.21 (official build))