clickhouse-operator: Unable to run distributed DDL

Describe the unexpected behaviour When I execute a statement with ON CLUSTER, It should be able to run it on all the shards and replicas. However it is timing out.

How to reproduce

• Which ClickHouse server version to use: 21.12.3.32
• Which interface to use, if matters: Clickhouse-operator
• CREATE TABLE statements for all tables involved:

CREATE TABLE events_local on cluster '{cluster}' ( 
    event_date  Date, 
    event_type  Int32, 
    article_id  Int32, 
    title       String 
) engine=ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/{database}/{table}', '{replica}') 
PARTITION BY toYYYYMM(event_date) 
ORDER BY (event_type, article_id);

• Queries to run that lead to unexpected result Above one and also create database statement: CREATE DATABASE test ON CLUSTER '{cluster}' ;

Expected behavior The statement should be executed and database/ table should be created successfully

Error message and/or stacktrace

Query id: 98c72107-9eab-40be-b56e-11dcefbf4e59



0 rows in set. Elapsed: 180.672 sec.

Received exception from server (version 21.12.3):
Code: 159. DB::Exception: Received from localhost:9000. DB::Exception: Watching task /clickhouse/repl-1s1r/task_queue/ddl/query-0000000003 is executing longer than distributed_ddl_task_timeout (=180) seconds. There are 1 unfinished hosts (0 of them are currently active), they are going to execute the query in background. (TIMEOUT_EXCEEDED)

Additional context https://github.com/ClickHouse/ClickHouse/issues/33019