nodejs-logging: Regular error showing Total timeout of API google.logging.v2.LoggingServiceV2 exceeded 600000 milliseconds

Hi, I’m also getting this error in google cloud function.

node v12, “@google-cloud/logging”: “^9.2.2”, “@google-cloud/logging-winston”: “^4.0.4”, pub/sub cloud function

The cloud function finished with status ‘ok’ at 18:40:27, and then it arises an Unhandled rejection and timeout error at 18:41:45

Seeing the same issue on multiple Cloud Run instances.

Using:

    "@google-cloud/error-reporting": "^2.0.4",
    "@google-cloud/logging": "^9.6.3",
    "@google-cloud/logging-winston": "^4.1.1",
    "winston": "^3.3.3"

• google-gax 2.29.1
• @grpc/grpc-js 1.4.4

If you’re seeing this issue often and you’re in a serverless environment like Cloud Run you should enable writing to stdout so that log writes aren’t async. This is documented here

One more thing to mention - if your code is running in serverless environment (e.g. Cloud Functions or Cloud Run), it is important to remember that serverless environment have only compute/memory resources for the duration of the request, so if you do something like: logger.log('foo'); rather than: await logger.log('foo'); you might get timeouts.

There are some alternatives which can be used:

1. logSync class is recommended for usage in serverless environment.
2. Another option for Cloud Functions or Cloud Run is to use JSON.stringify as described here.

There is a good blog post which provides tips for running Node.js in Cloud Funcrions/Cloud Run here.

Also seeing this error, using "@google-cloud/logging-winston": "^4.1.1". The app has internet connectivity at all times but still the error happens

having the same issue in my cloud function

Error: Total timeout of API google.logging.v2.LoggingServiceV2 exceeded 60000 milliseconds before any response was received.
    at repeat (/workspace/node_modules/google-gax/build/src/normalCalls/retries.js:66:31)
    at Timeout._onTimeout (/workspace/node_modules/google-gax/build/src/normalCalls/retries.js:101:25)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)

    "@google-cloud/logging": "^10.4.0",
    "@google-cloud/logging-winston": "^5.3.0",

My code uses a winston logger with LoggingWinston transport from “@google-cloud/logging-winston”:

export function createCloudLogger() {
  const cloudLogger = winston.createLogger({
    level: LOG_LEVEL,
    format: format.combine(
      format((info) => {
        info.trace = process.env.TRACE_ID;
        info[LOGGING_TRACE_KEY] = process.env.TRACE_ID;
        return info;
      })(),
    ),
    defaultMeta: getDefaultMetadataForTracing(),
    transports: [
      new LoggingWinston({
        projectId: process.env.GCP_PROJECT,
        labels: {
          component: <string>process.env.LOG_COMPONENT,
        },
        logName: "mylog",
        resource: {
          labels: {
            function_name: <string>process.env.K_SERVICE,
          },
          type: "cloud_function",
        },
        useMessageField: false,
        redirectToStdout: false,
      }),
    ],
  });
  return cloudLogger;
}

then I log using winston’s methods like logger.info, logger.verbose. The methods are sync, so there’s no await.

I get this error too if a request came to my app (in local development) when the internet was disconnected. This is obviously not likely to occur in production, but it is possibly unexpected behaviour for the app to crash whenever the logging library is unable to reach the GCP endpoint? (Especially given the ‘fire and forget’ approach recommended in the docs.)

This may be correct behaviour, but I assume a lot of users will want to set up some sort of global error catching for this particular error.