k8s-config-connector: SQLInstance Error 400: The incoming request contained invalid data., invalidRequest
Checklist
- I did not find a related open issue.
- I did not find a solution in the troubleshooting guide: (https://cloud.google.com/config-connector/docs/troubleshooting)
- If this issue is time-sensitive, I have submitted a corresponding issue with GCP support.
Bug Description
After updating the config connector operator to 1.122.0, I am now seeing the following error on all SQLInstance manifests:
Update call failed: error updating: updating SQLInstance my-sql-instance failed: googleapi: Error 400: The incoming request contained invalid data., invalidRequest
My SQLInstance manifests all look very similar to the following:
apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLInstance
metadata:
annotations:
cnrm.cloud.google.com/deletion-policy: abandon
cnrm.cloud.google.com/project-id: my-gcp-project
name: my-gcp-project-my-sql-instance
namespace: my-namespace
spec:
databaseVersion: POSTGRES_16
region: us-west3
resourceID: my-sql-instance
settings:
availabilityType: REGIONAL
backupConfiguration:
enabled: true
pointInTimeRecoveryEnabled: true
databaseFlags:
- name: cloudsql.iam_authentication
value: 'on'
diskSize: 100
ipConfiguration:
requireSsl: true
maintenanceWindow:
day: 1
hour: 4
updateTrack: stable
tier: db-custom-2-8192
Additional Diagnostic Information
I have not been able to find any other logs that point to what the issue could be. That said, I have this same issue for 4 different SQLInstance manifests.
Kubernetes Cluster Version
v1.30.3-gke.1639000
Config Connector Version
1.122.0
Config Connector Mode
namespaced mode (default)
Log Output
{“SQLInstance”:{…}, “controller”:“sqlinstance-controller”, “controllerGroup”:“sql.cnrm.cloud.google.com”, “controllerKind”:“SQLInstance”, “error”:“Update call failed: error updating: updating SQLInstance my-sql-instance failed: googleapi: Error 400: The incoming request contained invalid data., invalidRequest”, “msg”:“Reconciler error”, “name”:“my-gcp-project-my-sql-instance”, “namespace”:“my-namespace”, “reconcileID”:“f7964001-890b-478f-80e7-67d9f2262cd5”, “timestamp”:“2024-09-16T22:59:48.295Z”}
Steps to reproduce the issue
Apply a SQLInstance manifest.
YAML snippets
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 23
Ah, no worries. Thanks @PurseChicken for the reply. Just wanted to confirm you aren’t seeing any “request contained invalid data” errors for the SQLInstances which have enabled the new direct controller annotation, and that they are working as intended.
@PurseChicken that is strange… Do you mind posting an example log line for one of the resources that shows “Running reconcile”? And, also an example log for one of the SQLInstances that has the annotation (that should be showing “Running reconcile”, but instead shows “starting reconcile”) ?
For me, here is an example of a log line for a SQLInstance that has the annotation:
And, here is an example of a log line for a SQLInstance that does not have the annotation:
@PurseChicken that’s great! Thank you.
In v1.124.0, it’s not obvious which controller is running. We plan to improve this in the future. For now, you can tell which controller is active due to some slight differences in the logging. For the legacy terraform-based controller, you will see a log message that says
"starting reconcile"when reconciling the resource. For the new direct controller, the log will show"Running reconcile"instead.@PurseChicken no worries, really appreciate you taking the time to give some feedback and try it out.
@jasonvigil FYI I was able to go to v1.124.0 without issue.
I just need to modify my helm chart for ConfigConnector so I can add the annotation and then I will report back on that. Thanks for your patience.
@jasonvigil I will first update to v1.124.0 and make sure that everything looks good. After that I will try the new controller and report back my findings.
Ok thank you @PurseChicken – I think we found the issue, it’s a bad release (for SQLInstance). Working on a fix now.