dask-cuda: LocalCUDACluster doesn't use multiple GPUs when cuDF is import
Only the first GPU is used when importing cudf. If I remove that import, all GPUs get assigned work to do.
from dask_cuda import LocalCUDACluster
from dask.distributed import Client
import numpy as np
import cupy
import dask
import dask.array as da
import cudf
if __name__ == '__main__':
cluster = LocalCUDACluster()
client = Client(cluster)
x = cupy.random.random((100000, 1000))
d = da.from_array(x, chunks=(10000, 1000), asarray=False)
u, s, v = np.linalg.svd(d)
s, v = dask.compute(s, v)
Obviously, this example is not really using cudf for anything, just serving for demonstration purposes.
The only workaround I found for this is to start dask-scheduler and dask-cuda-worker outside of my python script. Am I missing something or doing something wrong?
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 17 (14 by maintainers)
@VibhuJawa you’re not importing
cudfdirectly, but you are importing it indirectly when youimport dask_cudf. I think this is where the context is coming from.