nodejs-pubsub: (Exactly once delivery) BatchError: Failed to "modifyAckDeadline" Reason: 3 INVALID_ARGUMENT.

1. Is this a client library issue or a product issue? Client Library Issue.

2. Did someone already solve this?

• Search the issues already opened: https://github.com/googleapis/nodejs-pubsub/issues
• Search the issues on our “catch-all” repository: https://github.com/googleapis/google-cloud-node
• Search or ask on StackOverflow (engineers monitor these tags): http://stackoverflow.com/questions/tagged/google-cloud-platform+node.js

1. Do you have a support contract? No.

If the support paths suggested above still do not result in a resolution, please provide the following details.

Environment details

• OS: Mac OS Monterey 12.3.1 (M1)
• Node.js version: v14.19.1
• npm version: 6.14.16
• @google-cloud/pubsub version: ^2.19.0

Steps to reproduce

1. Create a Topic and Subscription for a project in the PubSub console. a) Assign the google API scope: gmail-api-push@system.gserviceaccount.com to the topic and subscription. b) Enable “Exactly once delivery” for the subscription.
2. Create a project which watches a mailbox for unread messages and gets the subscription just created:

const pubsub = new PubSub({ projectId: 'example-project' });
const sub = await pubsub.subscription('example-project-sub');

sub.on('error', async (error) => {
  console.info('Pubsub subscribe error: ', error);
});

sub.on('message', async (message) => {
  try {
    message.ack();
  } catch (err) {
    console.info(err);
  }
});

await gmail.users.watch({
    userId: 'me',
    requestBody: {
      labelIds: ['UNREAD'],
      topicName: `projects/example-project/topics/example-project-topic`,
    },
  })

(Authentication Code):

  const { google } = require('googleapis');

  const JWT = google.auth.JWT;
  const auth = new JWT({
    keyFile: 'credentials-service.json',
    scopes: ['https://www.googleapis.com/auth/gmail.readonly', 'https://mail.google.com/'],
    subject: 'email@gmail.com',
  });

  await auth.authorize();

  const gmail = google.gmail({
    auth,
    version: 'v1',
  });

1. Run the project concurrently with at least 2 instances.
2. Send 10 unique emails to the email matched to the topic in fast succession.
3. Witness the error appear:

BatchError: Failed to "modifyAckDeadline" for 1 message(s). Reason: 3 INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.
   at ./example-project/node_modules/@google-cloud/pubsub/src/message-queues.ts:258:15
   at async Promise.all (index 0)
   at async ModAckQueue._sendBatch (./example-project/node_modules/@google-cloud/pubsub/src/message-queues.ts:262:5)
   at async ModAckQueue.flush (./example-project/node_modules/@google-cloud/pubsub/src/message-queues.ts:142:7) {
 ackIds: [
   'BhYsXUZIUTcZCGhRDk9eIz81IChFEgADZ3kiHkYfYmhcXHUHUAUZei5hcmNbRwkIR1Z0X3MrFZ22qtYxUa3P8PXZdUh6FwULRVp_WV8ZC2tfX3YPUyWguYL5j-KZJBs-faaUqfgtLZu7y682Zi09WhJLLD5-NTZFQV5AEkw-A0RJUytDCypYEU4EISE-MD5FU0RQ'
 ],
 code: 3,
 details: 'Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.',
 metadata: Metadata {
   internalRepr: Map(2) {
     'google.rpc.errorinfo-bin' => "EXACTLY_ONCE_ACKID_FAILUREpubsub.googleapis.com�

�
BhYsXUZIUTcZCGhRDk9eIz81IChFEgADZ3kiHkYfYmhcXHUHUAUZei5iI2gPG1AIRlp0X3MrFcCaqtYxUa3P8PXZdUh6FwULRVp9X10TDWlVX3IGVCWguYL5j-KZJBs-ffu4qfgtLYDZlqg2Zi09WhJLLD5-NTZFQV5AEkw-A0RJUytDCypYEU4EISE-MD5FU0RQ PERMANENT_FAILURE_INVALID_ACK_ID",
     'grpc-status-details-bin' => "�
Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.�
(type.googleapis.com/google.rpc.ErrorInfo�
EXACTLY_ONCE_ACKID_FAILUREpubsub.googleapis.com�

�
BhYsXUZIUTcZCGhRDk9eIz81IChFEgADZ3kiHkYfYmhcXHUHUAUZei5iI2gPG1AIRlp0X3MrFcCaqtYxUa3P8PXZdUh6FwULRVp9X10TDWlVX3IGVCWguYL5j-KZJBs-ffu4qfgtLYDZlqg2Zi09WhJLLD5-NTZFQV5AEkw-A0RJUytDCypYEU4EISE-MD5FU0RQ PERMANENT_FAILURE_INVALID_ACK_ID"
   },
   options: {}
 }
}

Additional Details:

• There is no online record of this extended error log: "Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.".
  • Except for this PR in the pubsub .net repo.