ClickHouse: clickhouse 20.12.5 restart error: external dictionary not found

(you don’t have to strictly follow this form)

Describe the bug Reproduce of 12454.

When creating a dictionary and a materilized column, after restart we get external dictionary not found

Does it reproduce on recent release? 20.12.5

How to reproduce Create source table for dictionary

CREATE TABLE tests.ip_prefix (
  `prefix` String,
  `as_number` UInt32,
  `as_name` String
) ENGINE = MergeTree()
ORDER BY
  prefix SETTINGS storage_policy = 'ebs_gp2_jbod',
  index_granularity = 8192

Create dict

CREATE DICTIONARY ip_prefix_dict(
  prefix String DEFAULT '',
  as_number UInt32 DEFAULT 0,
  as_name String DEFAULT ''
)
PRIMARY KEY prefix
SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'default' TABLE 'ip_prefix' PASSWORD '' DB 'tests'))
LIFETIME(MIN 1 MAX 10)
LAYOUT(ip_trie)

Create table with Materilized Column

CREATE TABLE ip_test
(
    `timestamp`                         DateTime Codec (DoubleDelta, LZ4),
    `ip`                                String CODEC (LZ4),
    `is_ipv6`                           UINT8 MATERIALIZED isIPv6String(ip),
    `as_name`                           LowCardinality(String) MATERIALIZED dictGet('tests.ip_prefix_dict', 'as_name',
        tuple(if(is_ipv6, toIPv6(ip), toIPv4(ip))))
) ENGINE = MergeTree() PARTITION BY toYYYYMM(timestamp)
ORDER BY timestamp
SETTINGS
        index_granularity = 8192

Insert to ip_test works.

Now restart server and get error on unknown ip_prefix_dict

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 22 (4 by maintainers)

Most upvoted comments

@kitaisreal

We use dictGetFloat64 but ClickHouse doesn’t start and logs next trace:

2021.06.19 22:38:42.032634 [ 39 ] {} <Error> Application: Caught exception while loading metadata: Code: 36, e.displayText() = DB::Exception: external dictionary 'currencies' not found: While processing cast(dictGetFloat64('currencies', 'rate', tuple(toString(cast(toFixedString('USD', 3) AS currency_tmp_alter2931242594012237847, 'FixedString(3)') AS currency))) AS currency_rate_tmp_alter16863508477036305143, 'Float64') AS currency_rate: default expression and column type are incompatible.: Cannot attach table `stats`.`impressions_local` from metadata file /var/lib/clickhouse/metadata/stats/impressions_local.sql from query ATTACH TABLE stats.impressions_local (..., `currency` FixedString(3) DEFAULT toFixedString('USD', 3), `currency_rate` Float64 DEFAULT dictGetFloat64('currencies', 'rate', tuple(toString(currency)))) ENGINE = ReplicatedSummingMergeTree('/clickhouse/tables/ch2-{shard}/impressions_local', '{replica}', views) PARTITION BY toYYYYMM(date) PRIMARY KEY (...) ORDER BY (...) SETTINGS index_granularity = 8192: while loading database `stats` from path /var/lib/clickhouse/metadata/stats, Stack trace (when copying this message, always include the lines below):

0. DB::ExternalLoader::checkLoaded(DB::ExternalLoader::LoadResult const&, bool) const @ 0xec0ed5f in /usr/bin/clickhouse
1. std::__1::shared_ptr<DB::IExternalLoadable const> DB::ExternalLoader::load<std::__1::shared_ptr<DB::IExternalLoadable const>, void>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0xec0e8ae in /usr/bin/clickhouse
2. DB::FunctionDictHelper::getDictionary(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xa40f607 in /usr/bin/clickhouse
3. DB::FunctionDictHelper::getDictionary(DB::ColumnWithTypeAndName const&) @ 0xa40cca2 in /usr/bin/clickhouse
4. DB::FunctionDictGet<DB::DataTypeNumber<double>, DB::NameDictGetFloat64>::executeImpl(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0xa47a707 in /usr/bin/clickhouse
5. DB::IFunction::executeImplDryRun(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0xa0520fa in /usr/bin/clickhouse
6. DB::DefaultExecutable::executeDryRun(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0xa05145e in /usr/bin/clickhouse
7. DB::ExecutableFunctionAdaptor::executeWithoutLowCardinalityColumns(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0xa055636 in /usr/bin/clickhouse
8. DB::ExecutableFunctionAdaptor::defaultImplementationForConstantArguments(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0xa054e53 in /usr/bin/clickhouse
9. DB::ExecutableFunctionAdaptor::executeWithoutLowCardinalityColumns(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0xa0555f5 in /usr/bin/clickhouse
10. DB::ExecutableFunctionAdaptor::execute(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0xa055cf1 in /usr/bin/clickhouse
11. DB::ActionsDAG::addFunction(std::__1::shared_ptr<DB::IFunctionOverloadResolver> const&, std::__1::vector<DB::ActionsDAG::Node*, std::__1::allocator<DB::ActionsDAG::Node*> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool) @ 0xeb0a9cb in /usr/bin/clickhouse
12. DB::ActionsDAG::addFunction(std::__1::shared_ptr<DB::IFunctionOverloadResolver> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DB::Context const&) @ 0xeb09fa7 in /usr/bin/clickhouse
13. DB::ScopeStack::addFunction(std::__1::shared_ptr<DB::IFunctionOverloadResolver> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) @ 0xed41c15 in /usr/bin/clickhouse
14. DB::ActionsMatcher::visit(DB::ASTFunction const&, std::__1::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0xed475d7 in /usr/bin/clickhouse
15. DB::ActionsMatcher::visit(DB::ASTFunction const&, std::__1::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0xed4412b in /usr/bin/clickhouse
16. DB::InDepthNodeVisitor<DB::ActionsMatcher, true, std::__1::shared_ptr<DB::IAST> const>::visit(std::__1::shared_ptr<DB::IAST> const&) @ 0xed19907 in /usr/bin/clickhouse
17. DB::ExpressionAnalyzer::getRootActions(std::__1::shared_ptr<DB::IAST> const&, bool, std::__1::shared_ptr<DB::ActionsDAG>&, bool) @ 0xed1971c in /usr/bin/clickhouse
18. DB::ExpressionAnalyzer::getActionsDAG(bool, bool) @ 0xed24ea8 in /usr/bin/clickhouse
19. DB::validateColumnsDefaultsAndGetSampleBlock(std::__1::shared_ptr<DB::IAST>, DB::NamesAndTypesList const&, DB::Context const&) @ 0xf2ce212 in /usr/bin/clickhouse
20. DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, DB::Context const&, bool) @ 0xec342a4 in /usr/bin/clickhouse
21. DB::createTableFromAST(DB::ASTCreateQuery, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool) @ 0xebc29ca in /usr/bin/clickhouse
22. ? @ 0xebb827d in /usr/bin/clickhouse
23. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x8633b0d in /usr/bin/clickhouse
24. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()::operator()() @ 0x863606f in /usr/bin/clickhouse
25. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x8630f3d in /usr/bin/clickhouse
26. ? @ 0x8634af3 in /usr/bin/clickhouse
27. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
28. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
 (version 21.1.7.1 (official build))

Dictionary:

    <dictionary>
        <name>currencies</name>
        <source>
          <http>
            <url>...</url>
            <format>TabSeparated</format>
          </http>
        </source>
        <layout>
          <complex_key_hashed />
        </layout>
        <structure>
          <key>
            <attribute>
              <name>currency</name>
              <type>String</type>
            </attribute>
          </key>
          <attribute>
            <name>rate</name>
            <type>Float64</type>
            <null_value>1.0</null_value>
          </attribute>
        </structure>
        <lifetime>10800</lifetime>
    </dictionary>
+9
VKoptev on Jun 21, 2021
Read more comments on GitHub
← ClickHouse: CLI is broken starting with v22.3
ClickHouse: Clickhouse continuously generates error logs:"PostgreSQLReaplicaConsumer: connection error: Lost connection to the database server" →