ClickHouse: Unexpected behaviour RBAC

I try to use RBAC. Something happened on production. Grants to roles didn’t work correctly. I could reproduce this situation on test just once.

I created role and user on existing cluster:

create role test_role;
create user test_user identified by ‘test’;

And I granted select on existing table to role and role to user:

grant select on test1.table1 to test_role;
grant test_role to test_user;

Expected behavior: When i’ll connect as test_user I see result: select * from test1.table1;

But I received:

Received exception from server (version 21.2.10):
Code: 497. DB::Exception: Received from 127.0.0.1:9000. DB::Exception: test_user: Not enough privileges. To execute this query it's necessary to have grant SELECT(column1,column2,…) ON test1.table1. 

Revoke grants and grant again don’t give a result.

ClickHouse version 21.2.10

After some time it starts work correctly. Now everything is ok with others roles and users. Please can you explain what happened?

Staktrace:

2022.02.08 12:37:40.207539 [ 220451 ] {54440230-731f-4c37-ac35-16a6d8649d93} <Error> TCPHandler: Code: 497, e.displayText() = DB::Exception: test_user: Not enough privileges. To execute this query it's necessary to have grant SELECT(column1,column2,…) ON test1.table1, Stack trace:

0. bool DB::ContextAccess::checkAccessImpl2<true, false, std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >(DB::AccessFlags const&, std::__1::basic_string_view<char, std::__1::char_traits<char> > const&, std::__1::basic_string_view<char, std::__1::char_traits<char> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int)::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) const @ 0xe44c291 in /usr/bin/clickhouse
1. bool DB::ContextAccess::checkAccessImpl2<true, false, std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >(DB::AccessFlags const&, std::__1::basic_string_view<char, std::__1::char_traits<char> > const&, std::__1::basic_string_view<char, std::__1::char_traits<char> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) const @ 0xe44b0ac in /usr/bin/clickhouse
2. void DB::Context::checkAccessImpl<DB::AccessFlags, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >(DB::AccessFlags const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) const @ 0xe81776d in /usr/bin/clickhouse
3. DB::Context::checkAccess(DB::AccessFlags const&, DB::StorageID const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) const @ 0xe8174d3 in /usr/bin/clickhouse
4. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, std::__1::shared_ptr<DB::IBlockInputStream> const&, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&) @ 0xea231b9 in /usr/bin/clickhouse
5. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xea20efd in /usr/bin/clickhouse
6. DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::__1::shared_ptr<DB::IAST> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xed44a35 in /usr/bin/clickhouse
7. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xed43330 in /usr/bin/clickhouse
8. DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, DB::Context&, DB::SelectQueryOptions const&) @ 0xe9dd9e0 in /usr/bin/clickhouse
9. ? @ 0xeecf2db in /usr/bin/clickhouse
10. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool) @ 0xeecdebc in /usr/bin/clickhouse
11. DB::TCPHandler::runImpl() @ 0xf5ce1c5 in /usr/bin/clickhouse
12. DB::TCPHandler::run() @ 0xf5dde49 in /usr/bin/clickhouse
13. Poco::Net::TCPServerConnection::start() @ 0x11b7d8df in /usr/bin/clickhouse
14. Poco::Net::TCPServerDispatcher::run() @ 0x11b7f2f1 in /usr/bin/clickhouse
15. Poco::PooledThread::run() @ 0x11cb6659 in /usr/bin/clickhouse
16. Poco::ThreadImpl::runnableEntry(void*) @ 0x11cb24ba in /usr/bin/clickhouse
17. start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so
18. __clone @ 0xfe8dd in /usr/lib64/libc-2.17.so

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 3
  • Comments: 45 (26 by maintainers)

Most upvoted comments

I’m experiencing the same behavior here.

DB::Exception: svc_user Not enough privileges. To execute this query it's necessary to have grant CREATE TEMPORARY TABLE, REMOTE ON *.*. (ACCESS_DENIED)

SELECT enabledRoles()
['svc_role']


SELECT version()
22.3.4.20

SELECT *
FROM system.grants
WHERE role_name = 'svc_role'
LIMIT 2

┌─user_name─┬─role_name───────────┬─access_type────────────┬─database─┬─table─┬─column─┬─is_partial_revoke─┬─grant_option─┐
│ ᴺᵁᴸᴸ      │ svc_role │ CREATE TEMPORARY TABLE │ ᴺᵁᴸᴸ     │ ᴺᵁᴸᴸ  │ ᴺᵁᴸᴸ   │                 0 │            0 │
│ ᴺᵁᴸᴸ      │ svc_role │ REMOTE                 │ ᴺᵁᴸᴸ     │ ᴺᵁᴸᴸ  │ ᴺᵁᴸᴸ   │                 0 │            0 │
└───────────┴─────────────────────┴────────────────────────┴──────────┴───────┴────────┴───────────────────┴──────────────┘

SHOW GRANTS FOR svc_role

┌─GRANTS FOR svc_role───────────────────────────────────────┐
│ GRANT CREATE TEMPORARY TABLE, REMOTE ON *.* TO svc_user   │

I see this bug once in a while. Nothing helps but a clickhouse-server restart.

Can’t really reproduce this bug consistently. I have a group of servers with the same configuration. When I apply new grants to roles some of them work, some not.

It would be nice to have SYSTEM RELOAD ROLE CACHE or something.

Followed the script #34412 (comment) . No issues.

@m1khal3v , Please, provide clean and comprehensive repro.

I tried it and also couldn’t reproduce it locally, I apologize. The problem occurred on Yandex Cloud, but I did not try to reproduce it outside the cloud, perhaps this is some kind of exclusively cloud bug.

@ilejn, these are external LDAP users and groups, so default roles are not applicable to them.

And, as you see, enabledRoles() returns that alert-exporter-production-clickhouse-acl role is active.

Still reproducible on 22.8.17.17


clickhouse-01a :) show grants for `alert-exporter-production-clickhouse-acl`


┌─GRANTS FOR `alert-exporter-production-clickhouse-acl`─────────────────────────────────────┐
…
│ GRANT SHOW, SELECT, dictGet ON devops.* TO `alert-exporter-production-clickhouse-acl`     │
…
└───────────────────────────────────────────────────────────────────────────────────────────┘

clickhouse-01a :) show grants for svc_ch_alerter


┌─GRANTS FOR svc_ch_alerter─────────────────────────────────────────────────────────────────────────────────┐
│ GRANT `alert-exporter-production-clickhouse-acl`, `…` TO svc_ch_alerter │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────┘


clickhouse-01a :) select user()

┌─currentUser()──┐
│ svc_ch_alerter │
└────────────────┘

clickhouse-01a :) use devops

Received exception from server (version 22.8.17):
Code: 497. DB::Exception: Received from clickhouse-01a:9000. DB::Exception: svc_ch_alerter: Not enough privileges. To execute this query it's necessary to have grant SHOW DATABASES ON devops.*. (ACCESS_DENIED)