scylla-operator: 2 nodes in 7 node Scylla cluster missing in the middle of StatefulSet and we can't bootstrap new nodes there

What happened?

We originally had 7 nodes in our Scylla cluster, n2d-standard-32 with 3TB local SSDs, running Scylla 5.2.9, Scylla Operator 1.9.x, Scylla Manager 3.1.x.

We updated Scylla to 5.4.7, Scylla Operator 1.13.0 and Scylla Manager to 3.3.0 on the 25th of June. (Note that since that update we noticed https://github.com/scylladb/scylladb/issues/19793)

Then from June 25th to July 4th we migrated our Scylla from to a new node pool, with same machine size but a little different config - switching from the default SA to a custom one, used recommended oauth scopes. (Nothing that would affect Scylla directly, I think.)

On 12th of July we updated Scylla it to 5.4.9.

On July 19th there was a hardware issue on one GCP node with Scylla that caused it to be restarted and the local SSD contents were lost. This was pod 3 in the StatefulSet. The nodetool status after it happened was:

Datacenter: us-west1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns    Host ID                               Rack
UN  10.7.252.229  1.53 TB    256          ?       1ff1b8df-7a90-4321-a309-7cd69e20bd70  us-west1-b
UN  10.7.241.130  1.28 TB    256          ?       8a24c600-5525-490e-a3cd-314f6062d6a1  us-west1-b
UN  10.7.241.175  1.43 TB    256          ?       050dcc67-7bb8-4d5d-89b1-5dbe0bcbb8b2  us-west1-b
UN  10.7.241.174  1.43 TB    256          ?       05bb205f-9475-43b1-a609-c9846f55ee2f  us-west1-b
UN  10.7.249.238  1.43 TB    256          ?       b8f68c62-c462-4a30-a505-5ece9ae1ab0b  us-west1-b
UN  10.7.243.109  1.54 TB    256          ?       5ddc2724-001a-4f06-8987-47633577263f  us-west1-b
DN  10.7.248.124  1.35 TB    256          ?       15253b54-8f30-4583-b08e-469c10c58aa2  us-west1-b

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

…but when I did the node replace procedure it failed on:

ERROR 2024-07-19 15:55:51,956 [shard  0:main] init - Startup failed: std::runtime_error (Replaced node with Host ID 67322b56-9f99-4375-98d2-b14ce167bf7a not found)
2024-07-19 15:55:52,025 INFO exited: scylla (exit status 1; not expected)

(Note that the id from this message was not part of the cluster and some time later when did this procedure to remove the ghosts nodes from our cluster then that id was also NOT appearing anywhere then.)

Anyway, we decided to remove the node 15253b54-8f30-4583-b08e-469c10c58aa2 using procedure https://opensource.docs.scylladb.com/branch-5.4/operating-scylla/procedures/cluster-management/remove-node.html#removing-an-unavailable-node and then bootstrap the new one as a completely new node.

Node removal took ~13 hours but it succeeded.

But we didn’t succeed with bootstrapping a new node. We guessed that it’s because that although the nodetool status has shown 6 nodes then in the cluster, the ScyllaCluster object still had 7 of them:

Screenshot 2024-07-20 at 10 25 36 (1)

(Sorry for the screenshots with texts, that’s only way some of the states were preserved.)

Then we tried to replace node 3 with node 6 to then scale down the cluster “officially” by updating the values for ScyllaCluster to make it a 6 node cluster (we wanted to do that anyway, assuming that with our RF=3 this is a better way to have a more balancer cluster).

But then most probably we hit an issue with PV / PVC from the old GCP node being left attached to the pod 3 and probably because of that we weren’t able to start the replace procedure. 😦 (I think that this was why it happened because only yesterday I thought about this and today after did a cleanup of those PV / PVC I was able to make pod 3 schedule a new Scylla pod again.)

Ultimately we just left it as it was then but the next day we hit another issue: because of https://github.com/scylladb/scylladb/issues/19793 hit the limit of our 3TB local disk space and was killed. (Back then we didn’t yet know that we hit the 90% limit, not 100%, because we didn’t yet have the workaround from https://github.com/scylladb/scylla-operator/issues/2056).

This happened to pod 4 and we were afraid to try to restart it because we assumed that StatefulSet will not allow it to start until pod 3 is running.

So we had pod 3 unready and pod 4 not starting. Ultimately we hacked around StatefulSet to make it work - we created a copy of it, deleted it and oprhaning the pods, recreated it with podManagementPolicy: Parallel. The hack worked (we did this once in the past before to work around the STS limitation successfully), we managed to start pod 4 but it fell in a restart loop because it was getting its disk full quickly.

That was July 22nd already. We decided to start migrating our Scylla cluster to a new nodepool, with n2d-highmem-32 and 6TB disks - more memory to tune Scylla later to more recommended CPU to memory ratio and bigger disks to have more time to fix https://github.com/scylladb/scylladb/issues/19793.

The replace node worked this time and pod 4 was successfully moved to the new node pool.

In the meantime we did a cleanup of the ghosts nodes in our cluster, but we noticed some strangeness during it so we reported https://github.com/scylladb/scylladb/issues/20020. Our node ids and the ghosts are in that issue.

Anyway, we continued to migrate the remaining Scylla pods to the new node pool and it all worked well, we migrated all of them except the pod 0.

Note that pod 3 was still out, we didn’t fix that yet. We planned to look into it more after we complete the migration to the new node pool as the disk space was slowly running out on the nodes with 3TB disks.

So yesterday we started to migrate pod 0 according to the procedure but during the bootstrap, a few minutes after it started, we accidentally triggered a delete of its PVC. 😐 It was shown as terminating, so we were worried that we will wait for ~12-30 hours (this is how long it took for us recently) for the bootstrap only to get it terminated and all of that progress lost, so we manually deleted the pod, deleted the PVC and PV and resumed.

But then the new Scylla didn’t restart the bootstrap as expected.

Unfortunately because of the stress and of the conditions we worked in I don’t have the exact info right now what we did and in what order. 😦

I do have the logs from all Scyllas from that period, so I can query for things you want to know.

What I know is that:

  1. At some point we did see the node that we were trying to replace show in nodetool status with a null id, like in https://github.com/scylladb/scylladb/issues/19975
  2. We tried removing both the node we were trying to replace (1ff1b8df-7a90-4321-a309-7cd69e20bd70) as well as the ids of the nodes that were trying to replace it.
  3. We got this in Scylla logs when it was trying to replace the node:
ERROR 2024-08-08 17:18:21,500 [shard  0:main] init - Startup failed: std::runtime_error (Replaced node with Host ID 1ff1b8df-7a90-4321-a309-7cd69e20bd70 not found)
  1. We have tried setting, unsettling the node-replace label in the Services for 0 and 3. We tried removing the “internal” label there with the id of the node that is being replaces (hoping that the right one will be set on the next try).
  2. We have tried deleting the new GCP nodes that were being created for bootstrap to start with a clean disk.

So ultimately now we are left with both pod 3 and pod 0 not working and a cluster with this state:

root@gke-main-scylla-6-25fcbc5b-1mnq:/# nodetool status
Datacenter: us-west1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns    Host ID                               Rack
UN  10.7.241.130  1.44 TB    256          ?       787555a6-89d6-4b33-941c-940415380062  us-west1-b
UN  10.7.241.175  1.61 TB    256          ?       5342afaf-c19c-4be2-ada1-929698a4c398  us-west1-b
UN  10.7.241.174  1.34 TB    256          ?       813f49f9-e397-4d70-8300-79fa91817f11  us-west1-b
UN  10.7.249.238  1.35 TB    256          ?       5cc72b36-6fcf-4790-a540-930e544d59d2  us-west1-b
UN  10.7.243.109  1.37 TB    256          ?       880977bf-7cbb-4e0f-be82-ded853da57aa  us-west1-b

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

cqlsh> select host_id, up from system.cluster_status;

 host_id                              | up
--------------------------------------+------
 787555a6-89d6-4b33-941c-940415380062 | True
 5342afaf-c19c-4be2-ada1-929698a4c398 | True
 813f49f9-e397-4d70-8300-79fa91817f11 | True
 5cc72b36-6fcf-4790-a540-930e544d59d2 | True
 880977bf-7cbb-4e0f-be82-ded853da57aa | True

(5 rows)
cqlsh> select server_id,group_id from system.raft_state ;

 server_id                            | group_id
--------------------------------------+--------------------------------------
 5342afaf-c19c-4be2-ada1-929698a4c398 | 904c8960-2c68-11ee-979c-be9922839fd2
 5cc72b36-6fcf-4790-a540-930e544d59d2 | 904c8960-2c68-11ee-979c-be9922839fd2
 787555a6-89d6-4b33-941c-940415380062 | 904c8960-2c68-11ee-979c-be9922839fd2
 813f49f9-e397-4d70-8300-79fa91817f11 | 904c8960-2c68-11ee-979c-be9922839fd2
 880977bf-7cbb-4e0f-be82-ded853da57aa | 904c8960-2c68-11ee-979c-be9922839fd2
 aa434e8c-84a1-43a3-a398-28e4e6949e56 | 904c8960-2c68-11ee-979c-be9922839fd2
 ceb76652-0f39-4018-9fa5-dd8f0b25e85a | 904c8960-2c68-11ee-979c-be9922839fd2

(7 rows)

Now we can’t bootstrap the missing pods 0 and 3 as new nodes (as I guess this is the only option now, if the are no nodes to replace).

We are getting seemingly good info in the logs at first like:

INFO  2024-08-09 08:50:48,814 [shard  0:stre] gossip - Gossip shadow round finished with nodes_talked={10.7.241.130, 10.7.243.109, 10.7.249.238, 10.7.241.174, 10.7.241.175}

…but then we get a bunch of:

WARN  2024-08-09 08:39:10,120 [shard  0:stre] storage_service - bootstrap[c4ddd61b-3ecb-4cce-a17a-d6f5c4525f20]: Found pending node ops = {{10.7.241.130 -> {2214835c-54fa-4763-8fa8-4c9d421f7670}}, {10.7.243.109 -> {2214835c-54fa-4763-8fa8-4c9d421f7670}}, {10.7.241.174 -> {2214835c-54fa-4763-8fa8-4c9d421f7670}}, {10.7.249.238 -> {2214835c-54fa-4763-8fa8-4c9d421f7670}}, {10.7.241.175 -> {2214835c-54fa-4763-8fa8-4c9d421f7670}}}, sleep 5 seconds and check again

…and it ends with repeating:

INFO  2024-08-09 08:51:43,923 [shard  0:stre] gossip - (rate limiting dropped 497 similar messages) Waiting for 2 live nodes to show up in gossip, currently 1 present...

When we try to remove any (ghost) nodes now, we can’t:

root@gke-main-scylla-6-25fcbc5b-1mnq:/# nodetool removenode ceb76652-0f39-4018-9fa5-dd8f0b25e85a
nodetool: Scylla API server HTTP POST to URL '/storage_service/remove_node' failed: std::runtime_error (Operation removenode is in progress, try again)
See 'nodetool help' or 'nodetool help <command>'.

…but if we check what node is still being deleted, we see:

root@gke-main-scylla-6-25fcbc5b-1mnq:/# nodetool removenode status
RemovalStatus: No token removals in process.

Please let us know if you need any more info! Fixing this is the top priority for us.

What did you expect to happen?

Being now able to bootstrap the 2 missing nodes so we can restore our cluster to a fully healthy state.

How can we reproduce it (as minimally and precisely as possible)?

It’s really hard for me to say which steps of the above history were the main contributors to our current situation…

Scylla Operator version

1.13.0

Kubernetes platform name and version

$ kubectl version
$ kubectl version
Client Version: v1.29.7
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.6-gke.1254000

(Note: this is the current version. When the whole story started we probably had a little older version.)

Please attach the must-gather archive.

scylla-operator-must-gather-dn7mtkvkjwvg.zip

Anything else we need to know?

This was initiated in this Slack thread

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 31 (23 by maintainers)

Most upvoted comments

So your 6th node is joining (0 ordinal), and your ScyllaCluster has 7 desired replicas. Pod with 3 ordinal is missing and has a replace label. To bring back cluster back to expected state, I would suggest to remove both replace labels (scylla/replace: "" and internal.scylla-operator.scylladb.com/replacing-node-hostid: c5214c14-6fb6-4ade-b5c9-01bf9f5b2029) from Service scylla-us-west1-us-west1-b-3. Since you removed ghost nodes replace is no longer needed. Once -0 node joins the cluster, StatefulSet controller will recreate -3 Pod and it will start joining the cluster. Once last 7th node joins your cluster should be fully reconciled, and only then you may carry next topology changes if you still desire.

Make sure StatefulSet/ScyllaCluster replicas reflects your existing state of the cluster. Clear all replace labels from Services. Bring back OrderedReady podManagementPolicy. Then bump number of members in ScyllaCluster to expected value. It will create new pods via statefulset.