ClickHouse: insertions and merges fail due to "not enough space" exception

Describe what’s wrong The log file is growing fast and filling with many “not enough space” exceptions.

2022.06.12 16:44:27.101439 [ 61626 ] {} <Error> MergeFromLogEntryTask: virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 243. DB::Exception: Cannot reserve 1.03 GiB, not enough space. (NOT_ENOUGH_SPACE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb37173a in /usr/bin/clickhouse
1. DB::MergeTreeData::reserveSpacePreferringTTLRules(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, unsigned long, DB::MergeTreeDataPartTTLInfos const&, long, unsigned long, bool, std::__1::shared_ptr<DB::IDisk>) const @ 0x1632c4e3 in /usr/bin/clickhouse
2. DB::MergeFromLogEntryTask::prepare() @ 0x162b48d3 in /usr/bin/clickhouse
3. DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x164a3b28 in /usr/bin/clickhouse
4. DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x164a2f02 in /usr/bin/clickhouse
5. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(std::__1::shared_ptr<DB::TaskRuntimeData>) @ 0xb3489eb in /usr/bin/clickhouse
6. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::threadFunction() @ 0xb348639 in /usr/bin/clickhouse
7. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb41b7ca in /usr/bin/clickhouse
8. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xb41d964 in /usr/bin/clickhouse
9. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb418b97 in /usr/bin/clickhouse
10. ? @ 0xb41c71d in /usr/bin/clickhouse
11. start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so
12. __clone @ 0xfeb0d in /usr/lib64/libc-2.17.so
 (version 22.3.6.5 (official build))
2022.06.12 16:44:27.101492 [ 61624 ] {} <Error> MergeFromLogEntryTask: virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 243. DB::Exception: Cannot reserve 1.39 GiB, not enough space. (NOT_ENOUGH_SPACE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb37173a in /usr/bin/clickhouse
1. DB::MergeTreeData::reserveSpacePreferringTTLRules(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, unsigned long, DB::MergeTreeDataPartTTLInfos const&, long, unsigned long, bool, std::__1::shared_ptr<DB::IDisk>) const @ 0x1632c4e3 in /usr/bin/clickhouse
2. DB::MergeFromLogEntryTask::prepare() @ 0x162b48d3 in /usr/bin/clickhouse
3. DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x164a3b28 in /usr/bin/clickhouse
4. DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x164a2f02 in /usr/bin/clickhouse
5. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(std::__1::shared_ptr<DB::TaskRuntimeData>) @ 0xb3489eb in /usr/bin/clickhouse
6. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::threadFunction() @ 0xb348639 in /usr/bin/clickhouse
7. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb41b7ca in /usr/bin/clickhouse
8. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xb41d964 in /usr/bin/clickhouse
9. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb418b97 in /usr/bin/clickhouse
10. ? @ 0xb41c71d in /usr/bin/clickhouse
11. start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so
12. __clone @ 0xfeb0d in /usr/lib64/libc-2.17.so
 (version 22.3.6.5 (official build))
2022.06.12 16:44:27.101531 [ 61619 ] {} <Error> MergeFromLogEntryTask: virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 243. DB::Exception: Cannot reserve 1.44 GiB, not enough space. (NOT_ENOUGH_SPACE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb37173a in /usr/bin/clickhouse
1. DB::MergeTreeData::reserveSpacePreferringTTLRules(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, unsigned long, DB::MergeTreeDataPartTTLInfos const&, long, unsigned long, bool, std::__1::shared_ptr<DB::IDisk>) const @ 0x1632c4e3 in /usr/bin/clickhouse
2. DB::MergeFromLogEntryTask::prepare() @ 0x162b48d3 in /usr/bin/clickhouse
3. DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x164a3b28 in /usr/bin/clickhouse
4. DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x164a2f02 in /usr/bin/clickhouse
5. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(std::__1::shared_ptr<DB::TaskRuntimeData>) @ 0xb3489eb in /usr/bin/clickhouse
6. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::threadFunction() @ 0xb348639 in /usr/bin/clickhouse
7. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb41b7ca in /usr/bin/clickhouse
8. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xb41d964 in /usr/bin/clickhouse
9. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb418b97 in /usr/bin/clickhouse
10. ? @ 0xb41c71d in /usr/bin/clickhouse
11. start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so
12. __clone @ 0xfeb0d in /usr/lib64/libc-2.17.so
 (version 22.3.6.5 (official build))
2022.06.12 16:44:27.101578 [ 61617 ] {} <Error> MergeFromLogEntryTask: virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 243. DB::Exception: Cannot reserve 1.68 GiB, not enough space. (NOT_ENOUGH_SPACE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb37173a in /usr/bin/clickhouse
1. DB::MergeTreeData::reserveSpacePreferringTTLRules(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, unsigned long, DB::MergeTreeDataPartTTLInfos const&, long, unsigned long, bool, std::__1::shared_ptr<DB::IDisk>) const @ 0x1632c4e3 in /usr/bin/clickhouse
2. DB::MergeFromLogEntryTask::prepare() @ 0x162b48d3 in /usr/bin/clickhouse
3. DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x164a3b28 in /usr/bin/clickhouse
4. DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x164a2f02 in /usr/bin/clickhouse
5. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(std::__1::shared_ptr<DB::TaskRuntimeData>) @ 0xb3489eb in /usr/bin/clickhouse
6. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::threadFunction() @ 0xb348639 in /usr/bin/clickhouse
7. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb41b7ca in /usr/bin/clickhouse
8. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xb41d964 in /usr/bin/clickhouse
9. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb418b97 in /usr/bin/clickhouse
10. ? @ 0xb41c71d in /usr/bin/clickhouse
11. start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so
12. __clone @ 0xfeb0d in /usr/lib64/libc-2.17.so
 (version 22.3.6.5 (official build))
2022.06.12 16:44:27.101579 [ 61622 ] {} <Error> MergeFromLogEntryTask: virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 243. DB::Exception: Cannot reserve 1.19 GiB, not enough space. (NOT_ENOUGH_SPACE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb37173a in /usr/bin/clickhouse
1. DB::MergeTreeData::reserveSpacePreferringTTLRules(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, unsigned long, DB::MergeTreeDataPartTTLInfos const&, long, unsigned long, bool, std::__1::shared_ptr<DB::IDisk>) const @ 0x1632c4e3 in /usr/bin/clickhouse
2. DB::MergeFromLogEntryTask::prepare() @ 0x162b48d3 in /usr/bin/clickhouse
3. DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x164a3b28 in /usr/bin/clickhouse
4. DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x164a2f02 in /usr/bin/clickhouse
5. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(std::__1::shared_ptr<DB::TaskRuntimeData>) @ 0xb3489eb in /usr/bin/clickhouse
6. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::threadFunction() @ 0xb348639 in /usr/bin/clickhouse
7. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb41b7ca in /usr/bin/clickhouse
8. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xb41d964 in /usr/bin/clickhouse
9. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb418b97 in /usr/bin/clickhouse
10. ? @ 0xb41c71d in /usr/bin/clickhouse
11. start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so
12. __clone @ 0xfeb0d in /usr/lib64/libc-2.17.so
 (version 22.3.6.5 (official build))
2022.06.12 16:44:27.101616 [ 61625 ] {} <Error> MergeFromLogEntryTask: virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 243. DB::Exception: Cannot reserve 1.43 GiB, not enough space. (NOT_ENOUGH_SPACE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb37173a in /usr/bin/clickhouse
1. DB::MergeTreeData::reserveSpacePreferringTTLRules(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, unsigned long, DB::MergeTreeDataPartTTLInfos const&, long, unsigned long, bool, std::__1::shared_ptr<DB::IDisk>) const @ 0x1632c4e3 in /usr/bin/clickhouse
2. DB::MergeFromLogEntryTask::prepare() @ 0x162b48d3 in /usr/bin/clickhouse
3. DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x164a3b28 in /usr/bin/clickhouse
4. DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x164a2f02 in /usr/bin/clickhouse
5. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(std::__1::shared_ptr<DB::TaskRuntimeData>) @ 0xb3489eb in /usr/bin/clickhouse
6. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::threadFunction() @ 0xb348639 in /usr/bin/clickhouse
7. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb41b7ca in /usr/bin/clickhouse
8. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xb41d964 in /usr/bin/clickhouse
9. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb418b97 in /usr/bin/clickhouse
10. ? @ 0xb41c71d in /usr/bin/clickhouse
11. start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so
12. __clone @ 0xfeb0d in /usr/lib64/libc-2.17.so
 (version 22.3.6.5 (official build))

Does it reproduce on recent release? Clickhouse version: 21.8.15.7, 22.3.6.5. Platform: CentOS 7.9 x86_64

How to reproduce

Which ClickHouse server version to use
Which interface to use, if matters
Non-default settings, if any

Data path is changed to /data01/data/clickhouse. /data01 usage is only 1%.

[root@ck06 clickhouse-server]# df -lh                                                                                                   
文件系统                 容量  已用  可用 已用% 挂载点                                                                                  
devtmpfs                 126G     0  126G    0% /dev                                                                                    
tmpfs                    126G     0  126G    0% /dev/shm                                                                                
tmpfs                    126G  2.2G  124G    2% /run                                                                                    
tmpfs                    126G     0  126G    0% /sys/fs/cgroup                                                                          
/dev/mapper/centos-root  1.4T  143G  1.2T   11% /                                                                                       
/dev/nvme0n1p2          1014M  319M  696M   32% /boot                                                                                   
/dev/nvme0n1p1           200M   12M  189M    6% /boot/efi                                                                               
/dev/sda                  29T  123G   28T    1% /data01                                                                                 
tmpfs                     26G   12K   26G    1% /run/user/42                                                                            
tmpfs                     26G     0   26G    0% /run/user/1001                                                                          
tmpfs                     26G     0   26G    0% /run/user/0
[root@ck06 clickhouse-server]# df -lh -i
文件系统                Inode 已用(I) 可用(I) 已用(I)% 挂载点                                                                           
devtmpfs                  32M     633     32M       1% /dev
tmpfs                     32M       1     32M       1% /dev/shm                                                                         
tmpfs                     32M    1.7K     32M       1% /run
tmpfs                     32M      16     32M       1% /sys/fs/cgroup                                                                   
/dev/mapper/centos-root  137M    271K    136M       1% /                                            
/dev/nvme0n1p2           512K      37    512K       1% /boot
/dev/nvme0n1p1              0       0       0        - /boot/efi                                                                        
/dev/sda                 466M    104K    466M       1% /data01
tmpfs                     32M       9     32M       1% /run/user/42                                                                     
tmpfs                     32M       1     32M       1% /run/user/1001
tmpfs                     32M       1     32M       1% /run/user/0

ClickHouse: insertions and merges fail due to "not enough space" exception

About this issue

Most upvoted comments