k8s-config-connector: SQLInstance: reference SQLInstance X/Y is not ready
We observed the behaviour that all resources related to sql.cnrm.cloud.google.com/v1beta1 have multiple events where they state transition from Ready to DependencyNotReady. It seems that the state is flapping.
$ kubectl get sqldatabase -o json | jq '.items | map(.status.conditions)'
[
[
{
"lastTransitionTime": "2020-10-21T11:44:58Z",
"message": "reference SQLInstance X/Y is not ready",
"reason": "DependencyNotReady",
"status": "False",
"type": "Ready"
}
]
]
Sometime later:
kubectl get sqldatabase -o json | jq '.items | map(.status.conditions)'
[
[
{
"lastTransitionTime": "2020-10-21T11:44:58Z",
"message": "reference SQLInstance X/Y is not ready",
"reason": "DependencyNotReady",
"status": "False",
"type": "Ready"
}
]
]
Checking the events in the resource:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal UpToDate 24m (x63 over 22h) sqldatabase-controller The resource is up to date
Warning DependencyNotReady 4m48s (x855 over 22h) sqldatabase-controller reference SQLInstance X/Y is not ready
It happens for the rest of the resources related to CloudSQL, like sqlsslcert.sql.cnrm.cloud.google.com, sqldatabase.sql.cnrm.cloud.google.com, sqlinstance.sql.cnrm.cloud.google.com
Example from sqlsslcert.sql.cnrm.cloud.google.com
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal UpToDate 48m (x59 over 21h) sqlsslcert-controller The resource is up to date
Warning DependencyNotReady 7m59s (x829 over 21h) sqlsslcert-controller reference SQLInstance X/Y is not ready
Could you advise about this issue, please?
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 2
- Comments: 27 (8 by maintainers)
Hi @benwh, I apologize that we missed your question. Yes, you are correct: the controller should not be marking the resource
Ready: falsesimply to renew the lease. We agree that this is a bug, and we’ll work on fixing it.@eyalzek I’m going to close the issue for now since things seem like they’re working ok. Ping me on this thread if it’s still not fixed and I’ll re-open the issue.
@xiaobaitusi Just to clarify though, disabling conflict prevention shouldn’t really be required right? This still sounds like a bug in the controller if it’s not able to renew the lease on the resource without changing its
Readycondition tofalsetemporarily.We have this issue for most of our SQL instances. We’re currently on 1.34.0 but have seen this on multiple versions.
Same symptoms as above. Eventually all events balance out (e.g. 498 UpToDate and 498 Updating.) Based on the fact the
generationis the reconciliation loop is not ignoring the correct fields (like status) or something is legitimately changing.