ClickHouse: Some queries does not respect max_threads and can make server non-resposible by starting too many threads

Happens on official build.

Repro (tested on 23.7.4.5).

cat <<'EOF' | clickhouse-client -mn
DROP DATABASE IF EXISTS test_max_threads_leak;

CREATE DATABASE test_max_threads_leak;

CREATE TABLE test_max_threads_leak.test
(
    `a` String,
    `b` LowCardinality(String),
    `c` DateTime64(3),
    `val` String,
)
ENGINE = MergeTree
ORDER BY (a, b, c)
SETTINGS index_granularity = 8192, parts_to_delay_insert=0;

CREATE TABLE test_max_threads_leak.test_distributed
(
    `a` String,
    `b` LowCardinality(String),
    `c` DateTime64(3),
    `val` String,
)
ENGINE = Distributed('default', 'test_max_threads_leak', 'test', rand());

-- creating lot of parts, because the number of the threads clickhouse will start is proportional to the number of parts
SYSTEM STOP MERGES test_max_threads_leak.test;

INSERT INTO test_max_threads_leak.test (a, b, c, val) SELECT 'foo' as a, 'bar' as b, _CAST('1683504000', 'DateTime64') as c, 'baz' as val FROM numbers(4000) SETTINGS max_block_size = 1, min_insert_block_size_bytes=1,min_insert_block_size_rows=1;

select count() from system.parts where table = 'test' and active;
EOF

cat <<'EOF' | clickhouse-benchmark -i 250 -c 25 --max_threads=5 
SELECT DISTINCT 'val' AS fieldType, val AS value FROM test_max_threads_leak.test_distributed WHERE (a = 'foo') AND (value != '') AND (positionCaseInsensitiveUTF8(value, 'baz') > 0) AND (b = 'bar')  AND (c >= _CAST('1683504000', 'DateTime64')) ORDER BY c DESC LIMIT 5
EOF

clickhouse benchmark here starts 25 concurrent queries with max_threads=5. It should not use more than 125 threads. In practice, it tries to start more than 10000 and crashes with exceptions like

An error occurred while processing the query ...
 Code: 439. DB::Exception: Received from localhost:9000. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=238, jobs=238). Stack trace:

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000e91fc37 in /usr/bin/clickhouse
1. ? @ 0x000000000ea0cf46 in /usr/bin/clickhouse
2. ? @ 0x000000000ea0dc8e in /usr/bin/clickhouse
3. ? @ 0x000000000ea08546 in /usr/bin/clickhouse
4. DB::PipelineExecutor::spawnThreads() @ 0x00000000152d58f7 in /usr/bin/clickhouse
5. DB::PipelineExecutor::executeImpl(unsigned long) @ 0x00000000152d447e in /usr/bin/clickhouse
6. DB::PipelineExecutor::execute(unsigned long) @ 0x00000000152d4151 in /usr/bin/clickhouse
7. ? @ 0x00000000152e1f8d in /usr/bin/clickhouse
8. ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000ea070cf in /usr/bin/clickhouse
9. ? @ 0x000000000ea0d0c1 in /usr/bin/clickhouse
10. ? @ 0x00007fc84a0cc609 in ?
11. __clone @ 0x00007fc849ff1133 in ?
. (CANNOT_SCHEDULE_TASK) (version 23.7.4.5 (official build))

Also, the numbers reported in that error message are incorrect. Older versions (23.3) report it better:

Code: 439. DB::Exception: Received from localhost:9000. DB::Exception: Cannot schedule a task: no free thread (timeout=0) (threads=10000, jobs=10000).

The real threads usage can be checked by query

select event_time, CurrentMetric_GlobalThreadActive from system.metric_log where event_time > now() - 600 order by event_time ; 

Was observed after the upgrade to 23.3.