nodejs-bigtable: createReadStream didn't returns all data when stream is pause and resumed

Hello,

I have an issue regarding createReadStream method in nodejs bigtable client, in may situation I need to have a way to get only one item from the stream and do some process on it then get the next and so on, and I don’t want to store all the data in memory, and bigtable client in nodejs didn’t provide any thing to do it, I use the createReadStream and pause the stream once I get an Item/Row from the table.

This worked fine for small data, but when I tried to do it on large data about 500k rows I didn’t retrieve all the rows that I have in the table for the query I made (the ‘end’ event is emitted before I get all the result from the table).

My code use class similar to this code, the class is supposed to return promise resolves to the next item from the stream or undefined if the stream is finished

export default class Iterable {
    stream: any;
    buffer: any[] = [];
    public cnt: number = 0;
    id = 0;

    constructor(id: number, stream: any) {
        this.id = id;
        this.stream = stream;

        this.stream.on('end', () => {
            console.log('end', this.cnt, this.buffer.length);
            this.buffer.push(undefined);
        });
        this.stream.once('data', (d: any) => {
            this.buffer.push(d);
            this.stream.pause();
        });
    }

    async next(): Promise<any> {
        this.stream.resume();
        return new Promise(resolve => {
            if(this.cnt < this.buffer.length) {
                resolve(this.buffer[this.cnt++]);
            } else {
                let id = setInterval(() => {
                    if (this.cnt < this.buffer.length) {
                        clearInterval(id);
                        resolve(this.buffer[this.cnt++]);
                    }
                }, 1);
            }
        });
    }
}

I also try to use once to only listen on data event on time each time I call next() but I still also didn’t get all the data that I should get.

I really hope you solve this issue or create another way to get only one item from the bigtable at each step.

Environment details

  • OS: Ubuntu 19.10
  • Node.js version: 10.15.2
  • npm version: 6.12.0
  • @google-cloud/bigtable version: 2.3.1

I also use bigtable emulator

Steps to reproduce

  1. Have a big table with large number of rows
  2. Try to use the above class to get all the data

Thanks!

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 20 (8 by maintainers)

Most upvoted comments

@jlogsdon Thank you for the workaround update! As we will likely drop Node 12 (but continue to support Node 14+) we’ll still have to address this in the library (as per @igorbernstein2’s comment) but it’s great to know there’s an interim solution for users who are able to upgrade.

I am facing the same issue with the results being inconsistent. Most of the times the rows retrieved match the expected rows, but critically not always and not for all the streams that are running at the time. I have tried pull / push variations of the stream, as well as simple pull only (readable) stream implementation, all with the same results. Not sure if this is related but it happens when retrieving multiple streams, of more than 50K rows each, with the total processing running for more than 5 minutes. Also, the emulator is used, instead of an actual Bigtable instance. Any ideas on whether this is related to the client library or some other dependency (grpc?)?

I also tried using stream[Symbol.asyncIterator]() but failed with: (node:9665) UnhandledPromiseRejectionWarning: TypeError: this.stream[Symbol.asyncIterator] is not a function while using new Readable().wrap(stream)[Symbol.asyncIterator]() returns an async iterator that provides no results (after some delay).

Thanks

Hello

@stephenplusplus, I tried what we told me and I tried multiple ways including (pipe/unpine, cork/uncork and pause/resume) and I still didn’t successfully able to pause and resume the stream and get the correct data What I am try to Implement is iterator and when I want the row I will call function next (this behavior already implemented in java API for bigtable), so I need the item to be returned by calling next() in mentioned class thanks

Hello! Here’s what I’ve tried that works as expected:

const Transform = require('stream').Transform

const processStream = new Transform({
  objectMode: true,
  transform(row, enc, next) {
    setTimeout(() => {
      // Something async
      next(null, row)
    }, 1000)
  },
})

let numRowsReceived = 0

table.createReadStream()
  .pipe(processStream)
  .on('data', () => numRowsReceived++)
  .on('end', () => {
    // Verify numRowsReceived is expected number
    console.log('All results processed successfully!')
  })

The built-in Transform stream seems perfectly suited to what you’re after. Let me know if this will work for you.