TileDB-SOMA: Python regression: sparseNdArray.read_sparse_tensor slicing does not work

Recent regression likely related to C++ reader integration (#447): SparseNdArray.read_as_tensor no longer slices correctly. If coordinates (ids) are specified, it returns all rather than the requested slice. read_as_pandas_all() still works.

Example: I have a large 2D matrix. Using read_as_pandas_all with a slice produces a different result than read_sparse_tensor. They should be identical (in value)

In [21]: census['human'].ms['RNA'].X['raw'].read_as_pandas_all((0, slice(None)))
Out[21]: 
      soma_dim_0  soma_dim_1  soma_data
0              0          13        2.0
1              0          18        1.0
2              0          21        1.0
3              0          24        1.0
4              0          29        4.0
...          ...         ...        ...
4974           0       23715        7.0
4975           0       23719        1.0
4976           0       23724        1.0
4977           0       23729        3.0
4978           0       23745        1.0

[4979 rows x 3 columns]

In [22]: for i in census['human'].ms['RNA'].X['raw'].read_sparse_tensor((0, slice(None))): print(i)
<pyarrow.SparseCOOTensor>
type: float
shape: (176, 23747)

The (sliced) result above seems to be identical to the entire array, eg,

In [23]: for i in census['human'].ms['RNA'].X['raw'].read_sparse_tensor((slice(None), slice(None))): print(i)
<pyarrow.SparseCOOTensor>
type: float
shape: (176, 23747)

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 15 (15 by maintainers)

Commits related to this issue

Most upvoted comments

The one thing I definitely need for my use cases is the ability to call with multiple dimensions specified in a sequence (tuple usually, and this seems like the minimum necessary type with list as optional/nice-to-have).

Eg., (dim1coord, dim2coord, ...) Concrete example: ([0,3,4], slice(2,10))

Ideally, we would use semantics that are similar to numpy or TileDB-Py, which I understand as:

  • a sequence of coordinates is accepted
  • sequence length must be at least one and <= number of dimensions.
  • If the sequence contains missing coordinates (length less than number of dimensions), then “slice(None)” is assumed for the missing dimensions.
  • per-dimension, explicitly specified coordinates can be one of: a value, a list/ndarray/paarray/etc of values, a slice, etc.

So, for a dimension 3 array, ([1,2,3], ) is equivalent to ([1,2,3], slice(None), slice(None))

Did I miss any corner cases?

CC: @aaronwolen @thetorpedodog - in case you guys have any input on Pythonic conventions here.

This overall seems about right with regards to the way TileDB accepts coordinates. I know that there is some weirdness in the way that the APIs there deal with tuples as opposed to lists and how they are interpreted in terms of dimensions/ranges (like some_arr[a, b, c] is different from some_arr[[a, b, c]]?) but it has been a while since I dealt with that specific part of things. I would want to make sure we don’t effectively codify that exact behavior (since it seems like a very easy rake to step on).

@johnkerl - an related issue I found while testing this: if you specify an empty list/array for a dimension, it returns “all”. I expected it would return none, as this is what Numpy does. TileDB-Py on the other hand treats an empty coordinate list as all. Confusing.

Numpy-style:

In [26]: np.arange(10, dtype=np.float32)[[1]]
Out[26]: array([1.], dtype=float32)

In [27]: np.arange(10, dtype=np.float32)[[]]
Out[27]: array([], dtype=float32)

Our current implementation (SOMA and TileDB-Py both do this):

In [18]: census['human'].ms['RNA'].X['raw'].read_as_pandas_all([[],slice(None)])
Out[18]: 
        soma_dim_0  soma_dim_1  soma_data
0                0          13        2.0
1                0          18        1.0
2                0          21        1.0
3                0          24        1.0
4                0          29        4.0
...            ...         ...        ...
670301         175       23724        1.0
670302         175       23726        1.0
670303         175       23727        1.0
670304         175       23729        1.0
670305         175       23744        2.0

[670306 rows x 3 columns]

I’m not sure which is right. Personally, I think the NumPy model makes much more sense as it prevents the need for higher level error checking on joins.

Ideally, we would use semantics that are similar to numpy or TileDB-Py, which I understand as:

  • a sequence of coordinates is accepted
  • sequence length must be at least one and <= number of dimensions.
  • If the sequence contains missing coordinates (length less than number of dimensions), then “slice(None)” is assumed for the missing dimensions.
  • per-dimension, explicitly specified coordinates can be one of: a value, a list/ndarray/paarray/etc of values, a slice, etc.

So, for a dimension 3 array, ([1,2,3], ) is equivalent to ([1,2,3], slice(None), slice(None))

Did I miss any corner cases?

CC: @aaronwolen @thetorpedodog - in case you guys have any input on Pythonic conventions here.