ClickHouse: TCPHandler: Code: 432. DB::Exception: Unknown codec family code

After upgrade 19.5.3.8 to 21.8.15.7 (here we are not sure if we had problems) and then to 22.3.6.5 (we noticed problem with this version) we see the following traces in logs:

2022.07.18 12:53:01.101891 [ 549313 ] {540d8d64-9b10-4859-8125-b2ba5b535438} <Error> TCPHandler: Code: 432. DB::Exception: Unknown codec family code: 51: (while reading column event_date_time): (while reading from part /var/lib/clickhouse/data/kt_client_events/playback_events/20210620_20210630_2930997_3017132_75/ from mark 4332 with max_rows_to_read = 8192): While executing MergeTreeThread. (UNKNOWN_CODEC), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb37467a in /usr/bin/clickhouse
1. DB::CompressionCodecFactory::get(unsigned char) const @ 0x15039132 in /usr/bin/clickhouse
2. DB::CompressedReadBufferBase::readCompressedData(unsigned long&, unsigned long&, bool) @ 0x1502a3c3 in /usr/bin/clickhouse
3. DB::CompressedReadBufferFromFile::readBig(char*, unsigned long) @ 0x1502cba6 in /usr/bin/clickhouse
4. DB::SerializationNumber<unsigned int>::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const @ 0xb1c1dd3 in /usr/bin/clickhouse
5. DB::ISerialization::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::__1::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, COW<DB::IColumn>::immutable_ptr<DB::IColumn> > > >*) const @ 0x15213675 in /usr/bin/clickhouse
6. DB::MergeTreeReaderWide::readData(DB::NameAndTypePair const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, bool, unsigned long, unsigned long, std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, COW<DB::IColumn>::immutable_ptr<DB::IColumn> > > >&, bool) @ 0x1642c92f in /usr/bin/clickhouse
7. DB::MergeTreeReaderWide::readRows(unsigned long, unsigned long, bool, unsigned long, std::__1::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn> > >&) @ 0x1642b86b in /usr/bin/clickhouse
8. DB::MergeTreeRangeReader::DelayedStream::finalize(std::__1::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn> > >&) @ 0x16bf60ce in /usr/bin/clickhouse
9. DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::__1::deque<DB::MarkRange, std::__1::allocator<DB::MarkRange> >&) @ 0x16bfac20 in /usr/bin/clickhouse
10. DB::MergeTreeRangeReader::read(unsigned long, std::__1::deque<DB::MarkRange, std::__1::allocator<DB::MarkRange> >&) @ 0x16bf9415 in /usr/bin/clickhouse
11. DB::MergeTreeBaseSelectProcessor::readFromPartImpl() @ 0x16beec28 in /usr/bin/clickhouse
12. DB::MergeTreeBaseSelectProcessor::readFromPart() @ 0x16befd2d in /usr/bin/clickhouse
13. DB::MergeTreeBaseSelectProcessor::generate() @ 0x16bee4a0 in /usr/bin/clickhouse
14. DB::ISource::tryGenerate() @ 0x16907995 in /usr/bin/clickhouse
15. DB::ISource::work() @ 0x1690755a in /usr/bin/clickhouse
16. DB::SourceWithProgress::work() @ 0x16b5ee82 in /usr/bin/clickhouse
17. DB::ExecutionThreadContext::executeTask() @ 0x16927ce3 in /usr/bin/clickhouse
18. DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) @ 0x1691b73e in /usr/bin/clickhouse
19. ? @ 0x1691d0a4 in /usr/bin/clickhouse
20. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb41bad7 in /usr/bin/clickhouse
21. ? @ 0xb41f65d in /usr/bin/clickhouse
22. ? @ 0x87887 in /lib64/libc.so.6
23. ? @ 0x10993c in /lib64/libc.so.6

We’ve tried to upgrade to a recent release, 22.3.8.39, but it didn’t help.

How to reproduce

Currently, we don’t know.

Additional context

Is it safe to downgrade on 19.5.3.8 or even 18.12.17 to check if problem persists on old versions?