nodejs-logging-winston: Library still throws if we try to log a large object

Forgive my directness, but what’s the point of closing an issue on the basis of not yet having a viable solution to it, only to have to open a new issue later, isn’t the entire purpose of logging the issue to track the identification and resolution?

Leaving the issue open to retain visibility of the problem seems like a reasonable approach in such a situation, no?

In my case, to solve the issue I created a wrapper for the LoggingWinston with a custom truncate logic before send the message to the logger.

const TransportStream = require('winston-transport');
const { LoggingWinston } = require('@google-cloud/logging-winston')

class wrapperGoogleLogsTransport extends TransportStream {
  constructor(opts) {
    super(opts)
    this.googleLogsTransport = new LoggingWinston(opts)
  }

  log(info, callback) {
   // here you can write your custom logic, like truncate or maybe chunk info in parts and send by part
    const sizeInfo = JSON.stringify(info).length
    if (sizeInfo > 256000) {
      info = JSON.stringify(info).substring(0, 256000)
    }
    this.googleLogsTransport.log(info, callback)
  }
}

export default wrapperGoogleLogsTransport

A while ago, I did a fix in base nodejs-logging library to support more fields for truncation - here is a PR 1177. Thats said, I never exposed this logic in nodejs-logging-winston - please let me know if this is something you would like me to do. In addition, recently I changed the logic to include partialSuccess to be true by default - this at least make sure that only oversized entries will be dropped from request (if of course the size of the request itself does not goes beyond limit as specified in log-limits) - in this case while the error still would be returned, in response it will indicate which entries were dropped. Also, I believe we should implement log splitting as mentioned in LogSplit - this is a feature we can work on if there will be enough demand

I’d use that @losalex !

Could you propose an acceptable algorithm that might work the way we’ed need on objects? I’m not sure if there is a generic enough computationally viable solution to this.

like data could be preserved based on how close it is to the top of the json object. doing a shallow first count of bytes. Then arbitrary keys at a certain depth would be truncated after bytes are exhausted. It’s hard to say this is what every user will expect/