TileDB-SOMA: Python regression: sparseNdArray.read_sparse_tensor slicing does not work
Recent regression likely related to C++ reader integration (#447): SparseNdArray.read_as_tensor no longer slices correctly. If coordinates (ids) are specified, it returns all rather than the requested slice. read_as_pandas_all() still works.
Example: I have a large 2D matrix. Using read_as_pandas_all with a slice produces a different result than read_sparse_tensor. They should be identical (in value)
In [21]: census['human'].ms['RNA'].X['raw'].read_as_pandas_all((0, slice(None)))
Out[21]:
soma_dim_0 soma_dim_1 soma_data
0 0 13 2.0
1 0 18 1.0
2 0 21 1.0
3 0 24 1.0
4 0 29 4.0
... ... ... ...
4974 0 23715 7.0
4975 0 23719 1.0
4976 0 23724 1.0
4977 0 23729 3.0
4978 0 23745 1.0
[4979 rows x 3 columns]
In [22]: for i in census['human'].ms['RNA'].X['raw'].read_sparse_tensor((0, slice(None))): print(i)
<pyarrow.SparseCOOTensor>
type: float
shape: (176, 23747)
The (sliced) result above seems to be identical to the entire array, eg,
In [23]: for i in census['human'].ms['RNA'].X['raw'].read_sparse_tensor((slice(None), slice(None))): print(i)
<pyarrow.SparseCOOTensor>
type: float
shape: (176, 23747)
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 15 (15 by maintainers)
Commits related to this issue
- Bugfix #479 per se — committed to single-cell-data/TileDB-SOMA by johnkerl 4 years ago
- Bugfix #479 per se — committed to single-cell-data/TileDB-SOMA by johnkerl 4 years ago
- Update slicing logic for SparseNdArray (#485) * Neaten index-handling logic following issue 470 / PR 483 * Bugfix #479 per se * Unit-test the bugfix per se * fix lint * Neaten index-handl... — committed to single-cell-data/TileDB-SOMA by johnkerl 4 years ago
- Update SOMAIndexedDataFrame indexing logic (#483) * Add error-handling in ids runtype type-resolution * Update indexing logic * Neaten index-handling logic following issue 470 / PR 483 * Bug... — committed to single-cell-data/TileDB-SOMA by johnkerl 4 years ago
The one thing I definitely need for my use cases is the ability to call with multiple dimensions specified in a sequence (tuple usually, and this seems like the minimum necessary type with list as optional/nice-to-have).
Eg.,
(dim1coord, dim2coord, ...)Concrete example:([0,3,4], slice(2,10))This overall seems about right with regards to the way TileDB accepts coordinates. I know that there is some weirdness in the way that the APIs there deal with tuples as opposed to lists and how they are interpreted in terms of dimensions/ranges (like
some_arr[a, b, c]is different fromsome_arr[[a, b, c]]?) but it has been a while since I dealt with that specific part of things. I would want to make sure we don’t effectively codify that exact behavior (since it seems like a very easy rake to step on).@johnkerl - an related issue I found while testing this: if you specify an empty list/array for a dimension, it returns “all”. I expected it would return none, as this is what Numpy does. TileDB-Py on the other hand treats an empty coordinate list as all. Confusing.
Numpy-style:
Our current implementation (SOMA and TileDB-Py both do this):
I’m not sure which is right. Personally, I think the NumPy model makes much more sense as it prevents the need for higher level error checking on joins.
Ideally, we would use semantics that are similar to numpy or TileDB-Py, which I understand as:
So, for a dimension 3 array,
([1,2,3], )is equivalent to([1,2,3], slice(None), slice(None))Did I miss any corner cases?
CC: @aaronwolen @thetorpedodog - in case you guys have any input on Pythonic conventions here.