ClickHouse: How to recreate the Clickhouse metadata in Zookeeper if all Zookeeper data is lost?

Problem

Clickhouse replica nodes data are still in the disk but all Zookeeper data in disk is gone (accidentally).

This has caused to prevent writing to the replicated tables. Reading from the replicated tables have no problem.

Question

Is there still a way to recover this? I tried creating the /var/lib/clickhouse/flags/force_restore_data and then restarted the Clickhouse (container), and the zookeeper metadata is still not created.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 26 (11 by maintainers)

Most upvoted comments

Yeah, you can rename them later. And you can backup existing data using freeze https://clickhouse.yandex/docs/en/query_language/alter/#alter_freeze-partition

And use parts from shadow folder.

Steps:

  1. create a new replicated table TEMP
  2. ALTER TABLE table_name FREEZE
  3. mv /var/lib/clickhouse/shadow/…table_name…/* /var/lib/clickhouse/db/TEMP/detached
  4. attach all parts / partitions
cd /var/lib/clickhouse/db/TEMP/detached
let i=1;for f in `ls -1` ; do echo $i $f;((i++)); echo "alter table TEMP attach part '$f';"|clickhouse-client ; done
  1. rename table_name to table_name_old
  2. rename TEMP to table_name
  3. rename table_name to table_name_old at replica
  4. create a new replicated table table_name at replica (it will download the data)

create a new replicated table table_name at replica (it will download the data)

you need to execute create table at replicas with the same zk path as zk path at the first replicas

@den-crane you mentioned just stop CH. I have data in the temp table (new). I restarted clickhouse and it won’t start, it’s getting an error DB::Exception: Cannot create table from metadata file /var/lib/clickhouse/metadata/default//clicks_sharded_tmp.sql, error: DB::Exception: Existing table metadata in ZooKeeper differs in index granularity byte. I had to delete the ZK path for it to be back up again. 😄

Current state:

  • I have the backups of clicks_sharded and the metadata sql file in a temporary directory.
  • clicks_sharded and clicks_sharded_old table does not exist now.
  • I have no ZK path in zookeeper because of the above.
  • I have clicks_sharded_tmp table with the data from attach partition from FREEZING.