TileDB-SOMA: [r] Non-iterated `soma_reader` should not return partial results

In soma_reader() we’ve been commanded to read the entire array at once. Currently we take one sip of the straw, and if that’s only a partial read, we print a warning but proceed:

https://github.com/single-cell-data/TileDB-SOMA/blob/123cb4370057eb47c5657ca2f846336a86e6032e/apis/r/src/rinterface.cpp#L144-L152

I submit that it’s the wrong behavior to return a partial array here, regardless of printing a warning (which might get lost in a long program output). We should do whatever’s necessary to read the whole array; if we run out of memory, that’s on the caller for having commanded the non-iterated read.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 16 (16 by maintainers)

Most upvoted comments

Yes rbind is that thing. The canonical (old school) use is do.call() to sweep and operation over a list so here do.call(rbind, result_list). There are various improvements, I happen to like dt <- rbindlist(list_of_dts) which scans the list of data.table objects first, computes total sizes, allocates once and injects. Faster and more efficient than the piece-by-piece approach. There may be something for Arrow table chunks too but if so I don’t know it yet.

The old / basic / first pass iterator I wrote does that (modulo not having native iterators …). See eg the old test in test_SOMAReader-Iterated.R. Oh, those test files are more candidates for the grande olde renaming part III…