scylla-rust-driver: scylla::transport::topology ... Could not fetch metadata ... Protocol Error: system.peers or system.local has invalid column type
scylla rust driver version: 0.13.0
scylla version: 2024.2.0~rc1-20240715.9bbb8c1483d7
Using latte stress tool we get following errors:
2745.003 6351 25404 0.243 0.535 0.659 0.859 1.187 2.099 2.441 2.771 4.829
2750.005 6350 25398 0.260 0.536 0.659 0.857 1.141 1.888 2.410 2.820 3.697
2755.005 6350 25400 0.267 0.558 0.692 0.920 1.313 2.136 2.523 3.396 10.486
[2m2024-07-28T08:53:36.013626Z[0m [33m WARN[0m [2mscylla::transport::topology[0m[2m:[0m Failed to fetch metadata using current control connection [3mcontrol_connection_address[0m[2m=[0m"10.142.0.59:9042" [3merror[0m[2m=[0mProtocol Error: system.peers or system.local has invalid column type
[2m2024-07-28T08:53:36.018797Z[0m [33m WARN[0m [2mscylla::transport::topology[0m[2m:[0m Failed to fetch metadata using current control connection [3mcontrol_connection_address[0m[2m=[0m"10.142.0.159:9042" [3merror[0m[2m=[0mProtocol Error: system.peers or system.local has invalid column type
[2m2024-07-28T08:53:36.025609Z[0m [33m WARN[0m [2mscylla::transport::topology[0m[2m:[0m Failed to fetch metadata using current control connection [3mcontrol_connection_address[0m[2m=[0m"10.142.0.142:9042" [3merror[0m[2m=[0mProtocol Error: system.peers or system.local has invalid column type
[2m2024-07-28T08:53:36.030338Z[0m [33m WARN[0m [2mscylla::transport::topology[0m[2m:[0m Failed to fetch metadata using current control connection [3mcontrol_connection_address[0m[2m=[0m"10.142.0.125:9042" [3merror[0m[2m=[0mProtocol Error: system.peers or system.local has invalid column type
[2m2024-07-28T08:53:36.035767Z[0m [33m WARN[0m [2mscylla::transport::topology[0m[2m:[0m Failed to fetch metadata using current control connection [3mcontrol_connection_address[0m[2m=[0m"10.142.0.154:9042" [3merror[0m[2m=[0mProtocol Error: system.peers or system.local has invalid column type
[2m2024-07-28T08:53:36.041411Z[0m [33m WARN[0m [2mscylla::transport::topology[0m[2m:[0m Failed to fetch metadata using current control connection [3mcontrol_connection_address[0m[2m=[0m"10.142.0.121:9042" [3merror[0m[2m=[0mProtocol Error: system.peers or system.local has invalid column type
2760.001 6350 25400 0.275 0.575 0.712 0.940 1.351 2.169 2.460 2.793 6.042
[2m2024-07-28T08:53:36.048046Z[0m [33m WARN[0m [2mscylla::transport::topology[0m[2m:[0m Failed to fetch metadata using current control connection [3mcontrol_connection_address[0m[2m=[0m"10.142.0.131:9042" [3merror[0m[2m=[0mProtocol Error: system.peers or system.local has invalid column type
[2m2024-07-28T08:53:36.055162Z[0m [33m WARN[0m [2mscylla::transport::topology[0m[2m:[0m Failed to establish control connection and fetch metadata on all known peers. Falling back to initial contact points.
...
[2m2024-07-28T09:13:28.302729Z[0m [33m WARN[0m [2mscylla::transport::topology[0m[2m:[0m Failed to fetch metadata using current control connection [3mcontrol_connection_address[0m[2m=[0m"10.142.0.159:9042" [3merror[0m[2m=[0mProtocol Error: system.peers or system.local has invalid column type
[2m2024-07-28T09:13:28.305729Z[0m [31mERROR[0m [2mscylla::transport::topology[0m[2m:[0m Could not fetch metadata [3merror[0m[2m=[0mProtocol Error: system.peers or system.local has invalid column type
3952.774 6342 25367 0.259 0.577 0.693 0.852 1.031 1.137 1.328 1.516 2.378
It was not observed with previous versions of Scylla.
Ci job: https://jenkins.scylladb.com/job/enterprise-2024.2/job/longevity/job/longevity-gce-custom-d1-worklod2-hybrid-raid-test/2/ Argus: enterprise-2024.2/longevity/longevity-gce-custom-d1-worklod2-hybrid-raid-test#2
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 38 (1 by maintainers)
Ok, let me clear that up (you probably know most of that, but just to be sure we are on the same page). All the drivers have Load Balancing Policies (LBPs) - pieces of code that take a query and produce a list of nodes (+ optionally shards in case of our drivers) to send this query to - let’s call this a Query Execution Plan (or Plan). Why a list and not a single node? Because drivers have retry and speculative execution mechanisms, that make use of subsequent elements. Token Awareness (and Shard Awareness in our drivers) is implemented in LBPs - Token Aware LBP will put replicas for given query at the beginning of the plan. It’s similar for DC Awareness (and lately - Rack Awareness). User can configure LBP that is DC Aware by setting “local DC” - usually it will be the DC that the driver is running in, or a closest DC. Then such policy will prioritize (put on the beggining of Plan) the nodes that are in the “local DC”. There is one more piece of configuration for DC / Rack Aware LBPs: “dc failover”. If DISABLED, then Plan will only consist of nodes from “local DC”. If ENABLED, then nodes from “local DC” will be the first elements of the Plan, but then there will be nodes from other (“remote”) DCs.
The question is: if a query has CL=LOCAL*, should “dc failover” setting be ignored and treated as DISABLED?
@kbr-scylla I think the main question here is slightly different. If a user enabled DC failover (so is ok with requests being sent to different DC if they can’t be sent to preferred one), should the driver ignore this setting for requests with LOCAL* CL, and only send them to preferred DC? Because this is current behavior of Rust Driver and I’m not sure it is correct.
The argument for forcing CL=LOCAL* requests to always go to local DC is that we can achieve consistency (as defined by https://opensource.docs.scylladb.com/stable/cql/consistency-calculator.html for read CL and write CL = QUORUM - not sure what are the exact semantics, that reads will see all previous writes I think?) with regards to all clients connected to this DC. If we route the request outside of this DC then this guarantee no longer holds.
But there are counterarguments:
As I mentioned in the private message, Scylla documentation does not impose this requirements, and does not define LOCAL this way. From Scylla docs about
LOCAL_QUORUM:This is a strictly server-side definition and does not state any driver-side requirement - because consistency level is something to be handled server-side, not driver-side. I also don’t think any other driver impose such restrictions. If such restriction should be present, we should have a good reason for it, document it, and implement it in other drivers as well. But I suspect that the restriction should not be there and we should perform failover if user requested so.
Not sure who to ask here. @nyh ? @avikivity ?
Well, if a customer sets a particular consistency mode, we are obliged to either satisfy it or return an error. If a user expects local consistency, we mustn’t complete the query by having a remote node as the coordinator. There’s nothing about having a paying or nonpaying customer; it’s about having a customer with consistency guarantees fulfilled or broken.
also I think the missing part, is also that in a few seconds the data would be filled by scylla, i.e. on the next refresh.
as other driver doesn’t fails, and keeps on reading the peers table until correct data appear there.
I think it already agreed that rust driver is a bit too happy trigger to fail on missing data in peers tables compared to java/go/python.
Users would need to use this, and I suspect they won’t. It also introduces more cases (what if DC fetched from schema is different than provided one?).
I like the idea, but note that it would not help with this issue - because fetching peers failed. I think that instead of adding new things, we can make fetch not fail if one part of it fails (e.g. if schema fetching fails, but peers fetching succeeds, then we can return ClusterData with correct peers and empty schema).
Dup of https://github.com/scylladb/scylla-rust-driver/issues/1021 ?