tensorflow: tf.sparse.sparse_dense_matmul can't broadcast as tf.linalg.matmul does.
Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template
System information
- TensorFlow version (you are using): 2.1.0
- Are you willing to contribute it (Yes/No): No
Describe the feature and the current behavior/state.
tf.sparse.sparse_dense_matmul can’t broadcast on the leading dimensions. the following code is an example.
#!/usr/bin/python3
import numpy as np;
import tensorflow as tf;
def main():
a = tf.sparse.SparseTensor(
indices = [[0, 0, 1], [0, 0, 2], [0, 1, 2], [0, 1, 3], [0, 2, 1], [0, 2, 3]],
values = [1., 1., 1., 1., 1., 1.],
dense_shape = [1, 3, 4]
); # a.shape = (1,3,4)
b = tf.constant(np.random.normal(size = (4, 4, 5)), dtype = tf.float32);
c = tf.linalg.matmul(tf.sparse.to_dense(a),b); # will be succeed
c = tf.sparse.sparse_dense_matmul(a,b); # will be fail
if __name__ == "__main__":
main();
Will this change the current api? How?
no api need to be changed.
Who will benefit with this feature?
user handling sparse matrices will be benefit from the change.
Any Other info.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 17 (3 by maintainers)
please support matmul for sparse tensor. pytorch has deepspeed which can handle sparse tensor matmul having rank over 2. implementing sparse attention with tensorflow 2 is impossible without such feature.
In the code mentioned,
tf.rank(a).eval()is 3. The documentation says that the sparse multiplication works only for rank-2 tensors. Even in the traditional matmul function, it says thatSo, basically it is not supposed to work for matrices that are sparse, with rank > 2. IMO, it could work as the multiplication is performed on the last two dimensions only. So, I should handle the case where rank>2 in
sparse_matmuland perform multiplication only on the last two dimensions, instead of giving aValueError, right? This is my first contribution to the repository and I apologise for too many questions.