ClickHouse: segfault in simdjson while using JSON_VALUE

Describe what’s wrong

Version 22.6.1.1985 (official build), build id: A1A632F69545B4EC

I’m currently loading github archive in following table:

CREATE TABLE faster.github_events_raw
(
    `id` Int64,
    `ts` DateTime,
    `raw` String CODEC(ZSTD(16))
)
ENGINE = ReplacingMergeTree
PARTITION BY toYYYYMMDD(ts)
ORDER BY (ts, id)
SETTINGS index_granularity = 8192

I’ve got segmentation fault while doing following query:

SELECT DISTINCT JSON_VALUE(raw, '$.type') as t FROM faster.github_events_raw ORDER BY t;

Log:

2022.06.30 09:31:55.652437 [ 26320 ] <Fatal> BaseDaemon: ########################################
2022.06.30 09:31:55.655611 [ 26320 ] <Fatal> BaseDaemon: (version 22.6.1.1985 (official build), build id: A1A632F69545B4EC) (from thread 20935) (query_id: e48b0415-5ae3-4f98-b540-71e9228fbd68) (query: SELECT DISTINCT JSON_VALUE(raw, '$.type') as t FROM faster.github_events_raw ORDER BY t;) Received signal Segmentation fault (11)
2022.06.30 09:31:55.655747 [ 26320 ] <Fatal> BaseDaemon: Address: 0x7f4d547fe000 Access: read. Attempted access has violated the permissions assigned to the memory area.
2022.06.30 09:31:55.655928 [ 26320 ] <Fatal> BaseDaemon: Stack trace: 0x19a0cc66 0xeed21e1 0xef26d62 0xef26729 0xdf6566e 0x15374cd4 0x153756ba 0x15376b1c 0x15d51be5 0x17374e1c 0x12df9730 0x171904c7 0x171af9fa 0x171a491e 0x171a61e4 0xb94d0b7 0xb9504dd 0x7f4de6ae4b43 0x7f4de6b76a00
2022.06.30 09:31:55.693960 [ 26320 ] <Fatal> BaseDaemon: 2. simdjson::icelake::dom_parser_implementation::stage2(simdjson::dom::document&) @ 0x19a0cc66 in /usr/bin/clickhouse
2022.06.30 09:31:55.700939 [ 26320 ] <Fatal> BaseDaemon: 3. simdjson::dom::parser::parse_into_document(simdjson::dom::document&, unsigned char const*, unsigned long, bool) & @ 0xeed21e1 in /usr/bin/clickhouse
2022.06.30 09:31:55.704921 [ 26320 ] <Fatal> BaseDaemon: 4. DB::FunctionSQLJSONHelpers::Executor<DB::NameJSONValue, DB::JSONValueImpl, DB::SimdJSONParser>::run(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long, unsigned int) @ 0xef26d62 in /usr/bin/clickhouse
2022.06.30 09:31:55.705091 [ 26320 ] <Fatal> BaseDaemon: 5. DB::FunctionSQLJSON<DB::NameJSONValue, DB::JSONValueImpl>::executeImpl(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0xef26729 in /usr/bin/clickhouse
2022.06.30 09:31:55.706958 [ 26320 ] <Fatal> BaseDaemon: 6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0xdf6566e in /usr/bin/clickhouse
2022.06.30 09:31:55.707937 [ 26320 ] <Fatal> BaseDaemon: 7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x15374cd4 in /usr/bin/clickhouse
2022.06.30 09:31:55.708028 [ 26320 ] <Fatal> BaseDaemon: 8. DB::IExecutableFunction::executeWithoutSparseColumns(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x153756ba in /usr/bin/clickhouse
2022.06.30 09:31:55.708656 [ 26320 ] <Fatal> BaseDaemon: 9. DB::IExecutableFunction::execute(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x15376b1c in /usr/bin/clickhouse
2022.06.30 09:31:55.712713 [ 26320 ] <Fatal> BaseDaemon: 10. DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool) const @ 0x15d51be5 in /usr/bin/clickhouse
2022.06.30 09:31:55.713465 [ 26320 ] <Fatal> BaseDaemon: 11. DB::ExpressionTransform::transform(DB::Chunk&) @ 0x17374e1c in /usr/bin/clickhouse
2022.06.30 09:31:55.714151 [ 26320 ] <Fatal> BaseDaemon: 12. DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x12df9730 in /usr/bin/clickhouse
2022.06.30 09:31:55.714903 [ 26320 ] <Fatal> BaseDaemon: 13. DB::ISimpleTransform::work() @ 0x171904c7 in /usr/bin/clickhouse
2022.06.30 09:31:55.715626 [ 26320 ] <Fatal> BaseDaemon: 14. DB::ExecutionThreadContext::executeTask() @ 0x171af9fa in /usr/bin/clickhouse
2022.06.30 09:31:55.715731 [ 26320 ] <Fatal> BaseDaemon: 15. DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) @ 0x171a491e in /usr/bin/clickhouse
2022.06.30 09:31:55.715808 [ 26320 ] <Fatal> BaseDaemon: 16. ? @ 0x171a61e4 in /usr/bin/clickhouse
2022.06.30 09:31:55.716540 [ 26320 ] <Fatal> BaseDaemon: 17. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb94d0b7 in /usr/bin/clickhouse
2022.06.30 09:31:55.716653 [ 26320 ] <Fatal> BaseDaemon: 18. ? @ 0xb9504dd in /usr/bin/clickhouse
2022.06.30 09:31:55.716778 [ 26320 ] <Fatal> BaseDaemon: 19. ? @ 0x7f4de6ae4b43 in ?
2022.06.30 09:31:55.716856 [ 26320 ] <Fatal> BaseDaemon: 20. ? @ 0x7f4de6b76a00 in ?
2022.06.30 09:31:55.959984 [ 26320 ] <Fatal> BaseDaemon: Integrity check of the executable successfully passed (checksum: BC7D0D92B4030496529F48AA709DAAEE)

I was concurrently (n=80) ingesting data to this table in background. I will try to reproduce this with error reporting enabled.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 36 (23 by maintainers)

Commits related to this issue

Most upvoted comments

Hopefully you can quickly fix the issue now that we have a new release?

We had the same problem: our tests do not run typically on Ice Lake processors.

I am available to chat with anyone who would like to try out On Demand. It can be quite a bit faster in some cases! And it is well tested and documented.

@anvaari All versions >= v24.1 have the fix.

@anvaari This is not a segault as is the topic of this closed issue. The error you are reporting is Received signal Illegal instruction which is different. So I think you are encountering a different issue.

@alexey-milovidov

The ISA detection routine was updated in simdjson as of version 3.1.8 to check OS support for SIMD registers. Effectively, prior to 3.1.8, we assumed that if the CPU supported the ISA, we could run code with this ISA, but we did not account for OS-level disabling. I recommend updating the simdjson dependency in ClickHouse to the latest version of simdjson which is currently 3.6.3.

Note that simdjson is used in the Node.js runtime environment, among other places, so we have extensive testing in the wild and no bug report (so far). However, Node.js uses an up-to-date simdjson.

Maybe @vitlibar will be interesting in exploring the On Demand interface.

Please update simdjson to v2.2.0…

https://github.com/simdjson/simdjson/releases/tag/v2.2.0

It should fix the issue.

Can’t produce it locally on my Ryzen 5950x

Ryzen does not have AVX-512, and a different code path is used.

Thank you! We have received your Sentry report, btw 😃 Let’s find the offending JSONs.