initialization-actions: GPU utilization in spark error

I’m using attached install_gpu_driver.sh in dataproc 2.2. GPU is not getting recognized in spark. Attached installation logs for reference

dataproc-gpu-main.txt dataproc-initialization-script-0.log install_gpu_driver.txt

Command: Library: tensorflow[and-cuda] import tensorflow as tf print(tf.config.list_physical_devices(‘CPU’)) print(tf.config.list_physical_devices(‘GPU’))

Log: 2025-02-04 06:04:29.413125: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0. 2025-02-04 06:04:31.327492: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered WARNING: All log messages before absl::InitializeLog() is called are written to STDERR E0000 00:00:1738649071.978220 80542 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered E0000 00:00:1738649072.400870 80542 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2025-02-04 06:04:36.546724: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2025-02-04 06:04:50.776297: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:152] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error 2025-02-04 06:04:50.776349: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:137] retrieving CUDA diagnostic information for host: gpu-nvidia-l4-a363a292-a17a0463-m 2025-02-04 06:04:50.776359: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:144] hostname: gpu-nvidia-l4-a363a292-a17a0463-m 2025-02-04 06:04:50.776482: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:168] libcuda reported version is: 570.86.15 2025-02-04 06:04:50.776521: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:172] kernel reported version is: 570.86.15 2025-02-04 06:04:50.776531: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:259] kernel version seems to match DSO: 570.86.15 [PhysicalDevice(name=‘/physical_device:CPU:0’, device_type=‘CPU’)] []

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (11 by maintainers)

Most upvoted comments

oh, nice. they worked an F5 Networks, Inc. reference into the code. Good work, team little red ball. F5 still means acceleration.

case "${pci_device_id^^}" in
  ...
  "1EB8" ) gpu_type="nvidia-tesla-t4"   ;;
  "20B2" | \
  "20B5" | \
  "20F3" | \
  "20F5" ) gpu_type="nvidia-tesla-a100-80gb" ;;
  "20*"  ) gpu_type="nvidia-tesla-a100" ;;
  ...
esac

@cjac, after double-checking on the nvidia-l4, it turns out I hadn’t properly reviewed the logs earlier, and it is actually working fine. Sorry for the confusion.

Note also that there is a new, as-yet-undocumented feature which is still likely to change before being codified in the readme.

Passing the include-pytorch[1] metadata attribute value as yes will install pytorch and tensorflow to a conda environment. The name of the environment may be specified with the gpu-conda-env[2] metadata attribute.

[1] https://github.com/GoogleCloudDataproc/initialization-actions/blob/7e875225563b21680623ffc8bd04d8888aaef626/gpu/install_gpu_driver.sh#L1715 [2] https://github.com/GoogleCloudDataproc/initialization-actions/blob/7e875225563b21680623ffc8bd04d8888aaef626/gpu/install_gpu_driver.sh#L788

My guess is within the next two weeks.

I’d like to bring your attention to this section of the README.md, however:

https://github.com/GoogleCloudDataproc/initialization-actions?tab=readme-ov-file#how-initialization-actions-are-used


⚠️ NOTICE: For production usage, before creating clusters, it is strongly recommended that you copy initialization actions to your own Cloud Storage bucket to guarantee consistent use of the same initialization action code across all Dataproc cluster nodes and to prevent unintended upgrades from upstream in the cluster:

BUCKET=<your_init_actions_bucket>
CLUSTER=<cluster_name>
gsutil cp presto/presto.sh gs://${BUCKET}/
gcloud dataproc clusters create ${CLUSTER} --initialization-actions gs://${BUCKET}/presto.sh