longhorn: [BUG] Volumes stuck upgrading after 1.5.3 -> 1.6.0 upgrade.

I am not sure yet, but I think that may also be ineffective.

I cannot reproduce this behavior in my cluster and it is not seen by the upgrade tests in the CI either. It appears schema validation in the API server is rejecting these update requests, but I’m not sure why that would be the case in your cluster and not others. Are you aware of any cluster hardening you have in place that might affect this behavior?

The issue seems pretty similar to https://github.com/longhorn/longhorn/issues/3352, though that one is very old. The fix for it was to make some map fields nullable and to better ensure we submitted empty objects instead of nil to the Go Kubernetes client. Like the current issue, that one only seemed to be encountered in some clusters/versions.

This is my preferred attempt at a temporary fix. Unfortunately, I cannot validate it without being able to reproduce the issue.

kubectl edit crd instancemanagers.longhorn.io

Edit it in the following locations so that nullable: true:

              instanceEngines:
                additionalProperties:
                  properties:
                    spec:
                      properties:
                        backendStoreDriver:
                          description: 'Deprecated: Replaced by field `dataEngine`.'
                          type: string
                        dataEngine:
                          type: string
                        name:
                          type: string
                      type: object
                    status:
                      properties:
                        conditions:
                          additionalProperties:
                            type: boolean
                          nullable: true
                          type: object
						  
              instanceReplicas:
                additionalProperties:
                  properties:
                    spec:
                      properties:
                        backendStoreDriver:
                          description: 'Deprecated: Replaced by field `dataEngine`.'
                          type: string
                        dataEngine:
                          type: string
                        name:
                          type: string
                      type: object
                    status:
                      properties:
                        conditions:
                          additionalProperties:
                            type: boolean
                          nullable: true
                          type: object
						  
              instances:
                additionalProperties:
                  properties:
                    spec:
                      properties:
                        backendStoreDriver:
                          description: 'Deprecated: Replaced by field `dataEngine`.'
                          type: string
                        dataEngine:
                          type: string
                        name:
                          type: string
                      type: object
                    status:
                      properties:
                        conditions:
                          additionalProperties:
                            type: boolean
                          nullable: true
                          type: object

I think you should leave the nullable: true fields. It sounds like the preserveUnknownFields: false does not, at least, cause anything unexpectedly bad to occur, so we will likely include both changes in the next Longhorn release.

@Eilyre, I am pretty sure it will not cause any attach/detach operations, but it makes sense to me to wait until the other issue is resolved, just in case. I cannot easily test it because recent versions of Kubernetes won’t allow me to even set preserveUnknownFields: true as a means of reproducing the issue.

I think there’s an underlying problem that caused this problem, and is rearing it’s head on our cluster again @ejweber. Attach/detach/deletion operations are getting stuck again, but only when the volume needs to attach to another node. Volume creation goes through, but it won’t be able to attach to the pod.

The replicas stay in a weird state:

The errors in logs are very similar as before: Failed to get engine proxy of pvc-58297811-5dc4-4ac3-944d-056512745d8d-e-a4e533b1 for volume pvc-58297811-5dc4-4ac3-944d-056512745d8d, but there’s also a lot of Invalid gRPC metadata. I also do not understand why 4 replicas are kept for a lot of volumes - my default is configured to 3.

I sent a new version of support bundle.

@ejweber lets don’t forget to add issues to the upcoming milestone (1.7.0).

Should I make this a permanent patch for my environment, or do you think this is a one-off issue?

I would leave it for now. I suspect that we will want to make the change official in the next version of Longhorn, but I need to investigate a bit more. If you open any additional issues against Longhorn before this is fully resolved, please remind us of these changes up front so we can consider if they have an impact.

Also, are there any logs/information I could provide, if needed to figure out why this happens in some situations?

Not yet. I am going to experiment with your specific version of Kubernetes and see if I can trigger anything. Hopefully you will be available to answer some additional questions later on if I think of any?

Thanks for opening the issue!

Upgraded the system from version 1.5.3 to 1.6.0 today, to get around the RWX bug.

What issue number is that?

This one: https://github.com/longhorn/longhorn/issues/7183