ClickHouse: Anyway to manually fix Metadata on replica is not up to date with common metadata in Zookeeper?

I have killed ALTER query with DROP column as it took 2 days already. Now I can’t alter the table as it says

2020.11.26 18:47:14.354707 [ 11577 ] {04669f8b-7a8b-4ce8-a6c8-e458c263e847} <Error> DynamicQueryHandler: Code: 517, e.displayText() = DB::Exception: Metadata on replica is not up to date with common metadata in Zookeeper. Cannot alter, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x18e02790 in /usr/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0xe72fdad in /usr/bin/clickhouse
2. ? @ 0x1609f951 in /usr/bin/clickhouse
3. DB::InterpreterAlterQuery::execute() @ 0x15aa1723 in /usr/bin/clickhouse
4. ? @ 0x15e1ac62 in /usr/bin/clickhouse
5. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>) @ 0x15e1c9e9 in /usr/bin/clickhouse
6. DB::HTTPHandler::processQuery(DB::Context&, Poco::Net::HTTPServerRequest&, HTMLForm&, Poco::Net::HTTPServerResponse&, DB::HTTPHandler::Output&) @ 0x16466fb9 in /usr/bin/clickhouse
7. DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&) @ 0x1646aa6b in /usr/bin/clickhouse
8. Poco::Net::HTTPServerConnection::run() @ 0x18ce23e3 in /usr/bin/clickhouse
9. Poco::Net::TCPServerConnection::start() @ 0x18d205fb in /usr/bin/clickhouse
10. Poco::Net::TCPServerDispatcher::run() @ 0x18d20a8b in /usr/bin/clickhouse
11. Poco::PooledThread::run() @ 0x18e9f566 in /usr/bin/clickhouse
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x18e9a960 in /usr/bin/clickhouse
13. start_thread @ 0x74a4 in /lib/x86_64-linux-gnu/libpthread-2.24.so
14. __clone @ 0xe8d0f in /lib/x86_64-linux-gnu/libc-2.24.so
 (version 20.9.2.20 (official build))

Isn’t there really a way to fix metadata manually? I see what’s in ZK. Though I have not found anything in replica files to fix and the error message does not describe what is the issue exactly.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 25 (21 by maintainers)

Most upvoted comments

I would like to add one point here so anyone stumbling upon this thread might find useful

check your mutations and kill the stuck ones related to alter table as it really did the trick for me

select command,mutation_id from system.mutations where is_done=0

#Then kill all alter mutations that are stuck

kill mutation where command='<from above>'