TileDB-SOMA: [r] Non-iterated `soma_reader` should not return partial results
In soma_reader() we’ve been commanded to read the entire array at once. Currently we take one sip of the straw, and if that’s only a partial read, we print a warning but proceed:
I submit that it’s the wrong behavior to return a partial array here, regardless of printing a warning (which might get lost in a long program output). We should do whatever’s necessary to read the whole array; if we run out of memory, that’s on the caller for having commanded the non-iterated read.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 16 (16 by maintainers)
Yes rbind is that thing. The canonical (old school) use is
do.call()to sweep and operation over a list so heredo.call(rbind, result_list). There are various improvements, I happen to likedt <- rbindlist(list_of_dts)which scans the list ofdata.tableobjects first, computes total sizes, allocates once and injects. Faster and more efficient than the piece-by-piece approach. There may be something for Arrow table chunks too but if so I don’t know it yet.The old / basic / first pass iterator I wrote does that (modulo not having native iterators …). See eg the old test in
test_SOMAReader-Iterated.R. Oh, those test files are more candidates for the grande olde renaming part III…