pytorch_sparse: RuntimeError: CUDA error: an illegal memory access was encountered

The error seems to stem from the fact cuSPARSE cannot handle duplicated edges in edge_index. The reason for that is that it fails to compute the correct amount of output edges this way. In your case, it might well be that you have some initial self-loop edges in your graph, which should be removed before calling add_self_loops. I think your fix for augment_adj is correct, and I added it to the GraphUNet model in PyG.

@vthost @rusty1s Hi, I also met this error when use my own dataset to train Graph-UNet. This error randomly occurred when using GPU but never occurred when using CPU. I changed the augment_adj function, added the remove_self_loops function at first, and then the problem was solved. But I don’t know why.

def augment_adj(self, edge_index, edge_weight, num_nodes):
    edge_index, edge_weight = remove_self_loops(edge_index, edge_weight)
    edge_index, edge_weight = add_self_loops(edge_index, edge_weight, num_nodes=num_nodes)
    edge_index, edge_weight = sort_edge_index(edge_index, edge_weight, num_nodes)
    edge_index, edge_weight = spspmm(edge_index, edge_weight, edge_index, edge_weight, num_nodes, num_nodes, num_nodes)
    edge_index, edge_weight = remove_self_loops(edge_index, edge_weight)
    return edge_index, edge_weight

I don’t think that’s related to the above issue. You may have a memory leak somewhere, or one of your graphs in your dataset is too large that it can not be handled in a full-batch fashion.

I now have this also with ASAPool 😦

I am using ogbg-code. The example code for that data adds two types of edges to the graph in utils.augment_edge, so we might have several edges between two nodes. I tried to add coalesced=True in graph_unet.augment_adj as argument to spspmm but the error is still the same. It seems that spspmm interprets the coalesced argument as “sorted”. After I added the following in the beginning of graph_unet.forward (after the initialization of the edge weights), it runs for 74/143 epochs, and then the error comes again. If I add it in graph_unet.augment_adj, the training runs through, but I get the same error in the evaluation in remove_self_loops because the mask does not fit edge_attr[mask]. Just as an update… edge_index, edge_weight = coalesce(edge_index, edge_weight, x.shape[0], x.shape[0])

Can you show me an example code? For example, my GraphU-Net script runs just fine. Note that you need to pass coalesced=True if your edge_index is not sorted.