google-ads-java: Mutate Calls Internal Server Error : unexpected EOS on DATA frame

Hi, I am having a problem with the Java client library. I am creating lots of Campaigns, Adgroups, Ads, keywords, etc.

I use the Mutate for every resource. For example :

try (AdGroupAdServiceClient adGroupAdServiceClient = googleAdsClient.getLatestVersion().createAdGroupAdServiceClient()) {
                    MutateAdGroupAdsResponse response =
                            adGroupAdServiceClient.mutateAdGroupAds(Long.toString(customerId), adGroupAdOperations);
                    for (MutateAdGroupAdResult result : response.getResultsList()) {
                        System.out.printf(
                                "Expanded text ad created with resource name: %s%n", result.getResourceName());
                    }
                } catch (GoogleAdsException gae) {
                    int j = 0;
                    for (GoogleAdsError googleAdsError : gae.getGoogleAdsFailure().getErrorsList()) {
                        System.err.printf("  Error %d: %s%n", j++, googleAdsError);
                    }
                }

Before, i used to do this in just one call adding all the operations in the adGroupAdOperations (that could be up to 2k operations), but I started having a problem with an exception in GRPC. “unexpected EOS on DATA frame” . So, I started grouping this requests in about 100 operations per mutate call. Now i have the RESOURCE_TEMPORARILY_EXHAUSTED error. Is this the expected way to operate or is it some kind of bug regarding the GRPC client? If not, how can I know which is the right amount of operations in mutate calls not to fall in the EOS problem and sleep time between calls not to get the Resource Exhausted?

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 30 (16 by maintainers)

Most upvoted comments

I think you can only mutate the criteria for a single ad groups in a request. Our support staff on the forum would be best placed to help you with this.

A couple of quick suggestions that spring to mind:

Batch your operations together by ad group, rather than by product group.
BatchJobService would potentially take longer to execute, but I think would avoid this error.

But really, this is a question that’s better suited to our support teams. Please reach out to them on the forum and they’ll help you figure out a solution.

nwbirnie on Jan 21, 2021

So this request did fail, but the error in the logs is different:

errors: [{
    error_code: "AdGroupCriterionError.OPERATIONS_FOR_TOO_MANY_SHOPPING_ADGROUPS",
    field_path: "operations[12]",
    error_message: "Operations in the mutate request changes too many shopping ad groups. Please split requests for multiple shopping ad groups across multiple requests."
  },

Hopefully this gives you a heads up of what went wrong so you can make progress. In the meantime we’ll need to look at the grpc-java implemenation to see where the EOS error originates.

nwbirnie on Jun 26, 2019

Hey, this is a good question, thanks for asking.

Here are the relevant limits to bear in mind:

Maximum of 1000 mutate operations per request. (This will cause the EOS error.)
Maximum of 15k requests per day for Basic Access (no limit for Standard Access).
QPS is limited per CID, but varies depending on current load. So you should always handle RESOURCE_TEMPORARILY_EXHAUSTED and retry with an exponential backoff. If possible try to split your operations by CID, so if you have to add 10k keywords to 100 accounts, add 1000 at a time to each account, then the next 1000 and so on. If you have multiple threads, try keeping it below 10 threads at a time.

We are planning to launch a MutateJob service soon which will allow running larger workloads asynchronously and handle most/all of this on the backend. Stay tuned to the blog and release notes for updates!

nwbirnie on May 2, 2019