ClickHouse: A bug in Kafka engine: Segmentation fault (JSONEachRowRowInputFormat)

Kafka ingestion crashed the server. Data is serialized via jackson (possible missing field referenced by queue/topic table)

2021.01.18 16:19:09.348955 [ 30912 ] {} <Fatal> BaseDaemon: ########################################
2021.01.18 16:19:09.349037 [ 30912 ] {} <Fatal> BaseDaemon: (version 20.8.3.18, no build id) (from thread 4994) (no query) Received signal Segmentation fault (11)
2021.01.18 16:19:09.349063 [ 30912 ] {} <Fatal> BaseDaemon: Address: NULL pointer. Access: read. Address not mapped to object.
2021.01.18 16:19:09.349096 [ 30912 ] {} <Fatal> BaseDaemon: Stack trace: 0x1131d9c0 0x118459ac 0x112719d7 0x11890535 0x11891774 0x10a7675d 0x10a91b8a 0x11472f8d 0x11473988 0x1149cde2 0x1149d20a 0x1149d30f 0xa43cc4d 0xa43b3ff 0x7fa73043bea5 0x7fa72fd5896d
2021.01.18 16:19:09.349160 [ 30912 ] {} <Fatal> BaseDaemon: 3. DB::JSONEachRowRowInputFormat::readPrefix() @ 0x1131d9c0 in /usr/bin/clickhouse
2021.01.18 16:19:09.349189 [ 30912 ] {} <Fatal> BaseDaemon: 4. DB::IRowInputFormat::generate() @ 0x118459ac in /usr/bin/clickhouse
2021.01.18 16:19:09.349212 [ 30912 ] {} <Fatal> BaseDaemon: 5. DB::ISource::work() @ 0x112719d7 in /usr/bin/clickhouse
2021.01.18 16:19:09.349229 [ 30912 ] {} <Fatal> BaseDaemon: 6. ? @ 0x11890535 in /usr/bin/clickhouse
2021.01.18 16:19:09.349246 [ 30912 ] {} <Fatal> BaseDaemon: 7. DB::KafkaBlockInputStream::readImpl() @ 0x11891774 in /usr/bin/clickhouse
2021.01.18 16:19:09.349264 [ 30912 ] {} <Fatal> BaseDaemon: 8. DB::IBlockInputStream::read() @ 0x10a7675d in /usr/bin/clickhouse
2021.01.18 16:19:09.349289 [ 30912 ] {} <Fatal> BaseDaemon: 9. DB::copyData(DB::IBlockInputStream&, DB::IBlockOutputStream&, std::__1::atomic<bool>*) @ 0x10a91b8a in /usr/bin/clickhouse
2021.01.18 16:19:09.349307 [ 30912 ] {} <Fatal> BaseDaemon: 10. DB::StorageKafka::streamToViews() @ 0x11472f8d in /usr/bin/clickhouse
2021.01.18 16:19:09.349324 [ 30912 ] {} <Fatal> BaseDaemon: 11. DB::StorageKafka::threadFunc() @ 0x11473988 in /usr/bin/clickhouse
2021.01.18 16:19:09.349341 [ 30912 ] {} <Fatal> BaseDaemon: 12. DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x1149cde2 in /usr/bin/clickhouse
2021.01.18 16:19:09.349358 [ 30912 ] {} <Fatal> BaseDaemon: 13. DB::BackgroundSchedulePool::threadFunction() @ 0x1149d20a in /usr/bin/clickhouse
2021.01.18 16:19:09.349373 [ 30912 ] {} <Fatal> BaseDaemon: 14. ? @ 0x1149d30f in /usr/bin/clickhouse
2021.01.18 16:19:09.349395 [ 30912 ] {} <Fatal> BaseDaemon: 15. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xa43cc4d in /usr/bin/clickhouse
2021.01.18 16:19:09.349411 [ 30912 ] {} <Fatal> BaseDaemon: 16. ? @ 0xa43b3ff in /usr/bin/clickhouse
2021.01.18 16:19:09.349435 [ 30912 ] {} <Fatal> BaseDaemon: 17. start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so
2021.01.18 16:19:09.349459 [ 30912 ] {} <Fatal> BaseDaemon: 18. __clone @ 0xfe96d in /usr/lib64/libc-2.17.so

Thank you for your effort and time.

Regards, Ivan

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 26 (13 by maintainers)

Most upvoted comments

Hi, Confirmed - latest version works OK. We have not found any issues so far.

(version 21.1.2.15 (official build))

However migration from “20.8.3.18-1” to “21.1.2.15” needed some minor changes:

Default value (uuid field type) toUUID(0) was raising an error. toUUID(‘00000000-0000-0000-0000-000000000000’) solved the problem.

Thank you for your effort and time.

Regards, Ivan Prostran

ivanprostran on Jan 19, 2021

Thank you for the suggestions. Upgraded to newer version (for the moment without segmentation fault)

Now we have problem with a read only mode (all tables)

2021.01.19 09:55:23.538606 [ 30176 ] {} <Error> void DB::StorageKafka::threadFunc(size_t): Code: 242, e.displayText() = DB::Exception: Table is in readonly mode (zookeeper path: /clickhouse/tables/01-01/cm_downstream_quality_R), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x8692d81 in /usr/bin/clickhouse
1. DB::StorageReplicatedMergeTree::assertNotReadonly() const @ 0xf4386d9 in /usr/bin/clickhouse
2. DB::StorageReplicatedMergeTree::write(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::Context const&) @ 0xf4387f2 in /usr/bin/clickhouse
3. DB::PushingToViewsBlockOutputStream::PushingToViewsBlockOutputStream(std::__1::shared_ptr<DB::IStorage> const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::Context const&, std::__1::shared_ptr<DB::IAST> const&, bool) @ 0xee35528 in /usr/bin/clickhouse
4. DB::InterpreterInsertQuery::execute() @ 0xee293e1 in /usr/bin/clickhouse
5. DB::PushingToViewsBlockOutputStream::PushingToViewsBlockOutputStream(std::__1::shared_ptr<DB::IStorage> const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::Context const&, std::__1::shared_ptr<DB::IAST> const&, bool) @ 0xee34d54 in /usr/bin/clickhouse
6. DB::InterpreterInsertQuery::execute() @ 0xee293e1 in /usr/bin/clickhouse
7. DB::StorageKafka::streamToViews() @ 0xf4f41be in /usr/bin/clickhouse
8. DB::StorageKafka::threadFunc(unsigned long) @ 0xf4f346e in /usr/bin/clickhouse
9. DB::BackgroundSchedulePoolTaskInfo::execute() @ 0xeb39ae2 in /usr/bin/clickhouse
10. DB::BackgroundSchedulePool::threadFunction() @ 0xeb3bf17 in /usr/bin/clickhouse
11. ? @ 0xeb3d191 in /usr/bin/clickhouse
12. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x86415ed in /usr/bin/clickhouse
13. ? @ 0x86451a3 in /usr/bin/clickhouse
14. start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so
15. __clone @ 0xfe96d in /usr/lib64/libc-2.17.so
 (version 21.1.2.15 (official build))

Igore this (It was problem with zookeeper section not being referenced within config.d/config.xml)

metrika.xml:

	<zookeeper-servers>
		<node index="1">
			<host>localhost</host>
			<port>2181</port>
		</node>
	</zookeeper-servers>

config.d/config.xml:

  <remote_servers  incl="clickhouse_remote_servers"/>
   <zookeeper incl="zookeeper-servers" optional="true" />

Works now. Still checking segmentation fault occurences.

ivanprostran on Jan 19, 2021