external-dns: create/upsert records fail if too many AWS records to apply

restore a cluster backup will fail if you have too many records to create.

time="2018-10-06T23:44:37Z" level=error msg="InvalidChangeBatch: RDATA character limit of 32000 exceeded.\n\tstatus code: 400, request id: c855b089-c9c1-11e8-9fb5-cd7376597dff"
time="2018-10-06T23:44:37Z" level=error msg="Failed to submit all changes for the following zones: [/hostedzone/....]"

I think you need to do paged upsert/creates.

About this issue

Original URL
State: open
Created 6 years ago
Comments: 39 (17 by maintainers)

Most upvoted comments

We just got hit by this too.

It seems a bit weird to limit this by the number of records, when the limit is the size of the overall request. The correct number of records to limit to is dependent on the size of those records. This seems like something external-dns could calculate on its own, and split the request by size without needing a manual tunable.

alex-hunt-materialize on Mar 18, 2022

Added it as an extraarg in values file

ITBlogger on Dec 21, 2018

still an issue, maybe just set aws.batchChangeSize in helm chart to a lower value (currently is 1000), I synced fine with a value of 40

masterkain on Aug 16, 2022

/remove-lifecycle stale

tbarrella on Sep 14, 2019

I could workaround the problem with using --aws-batch-change-size=40 Maybe the default of 4000 is not a good setting.

szuecs on Oct 6, 2018