got: HTTPError: Response code 422 (Unprocessable Entity) upon second execution of pagination.paginate

Can you try the following:

1. set decompress to false
2. set methodRewriting to false

I tried each and both without any improveemnt.

If any of these are don’t fix the issue, please post your pagination function and after response hooks (there are two, right?)

Those might seem familiar for I made you sweat on it last week 😉

  pagination: {
    // eslint-disable-next-line @typescript-eslint/prefer-readonly-parameter-types
    paginate: (response: Response) => {
      const body: List = JSON.parse((response as Response<string>).body);
      if (is.plainObject(body) && is.number(body.next_offset) && body.next_offset >= 0) {
        const { options } = response.request;
        const { json, ...otherOptions }: any = options;
        const result: Options = {
          json: { ...json, offset: body.next_offset },
          ...otherOptions,
        };

        return result;
      }

      return false;
    },
    // eslint-disable-next-line @typescript-eslint/prefer-readonly-parameter-types
    transform: (response: Response<any>) => {
      const { body } = response;
      const parsed = is.string(body) ? JSON.parse(body) : body;
      if (is.plainObject(parsed)) {
        const { records } = parsed;
        if (is.array(records)) {
          return records as unknown[];
        }

        throw new Error(typeof records);
      }

      throw new Error(typeof parsed);
    },
  },

    afterResponse: [
      // eslint-disable-next-line @typescript-eslint/prefer-readonly-parameter-types
      async (response, retryWithMergedOptions) => {
        // Unauthorized
        if (response.statusCode === 401) {
          const updatedOptions: ExtendOptions = { headers: { 'oauth-token': await getNewAuthToken() } };

          // Save for further requests
          authenticatedInstance.defaults.options = got.mergeOptions(
            authenticatedInstance.defaults.options,
            updatedOptions
          );

          return retryWithMergedOptions(updatedOptions);
        }

        // No changes otherwise
        return response;
      },
    ],

(On my TODO list is a cleaner, more elegant and efficient management of OAuth2 token, including effective renewal)

That definitely seems to be a bug. It shouldn’t throw the mutually exclusive error if you use the body option only. I’ll look at the stack trace tomorrow.

I feel like I am being a pain here… 😉

So… this is in fact another symptom of #1223 root cause.

As soon as having pagination.paginate to return only the options update, the issue is fixed.

Thanks for your patience and time @szmarczak 😃

Here you can see two after response hooks. But in the code you provided there’s only one. Another bug here?

I need your input on this too 😃

Guess what? I have one single afterResponse hook.

Now if I inspect normalizedOptions.hooks.afterResponse in node_modules/got/dist/source/create.js@94, its length double at each iteration (i.e. 2 at page 2, 4 at page 3, 8 at page 4, etc.)

Funny for I was about to add my share to #1220 .

@szmarczak My idea seems to be correct.

I switched to using body in place of json and added a .padEnd(1024, ' ') to the stringified payload… no more crashes!

Thus I deduce that somehow the payload does indeed ends up being truncated upon requesting the second page.

I hope this will help.