detectron2: RuntimeError: CUDA error: invalid device function ROIAlign_forward_cuda

Attempting to forward inference the panoptic fpn model results in a CUDA error.

To Reproduce

Attempting to run a predictor using the model panoptic_fpn_R_101_dconv_cascade_gn_3x.yaml.

The following error is produced:

error in deformable_im2col: invalid device function
... < repeated ~30 times> ...
File "/opt/conda/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "/app/detectron2/detectron2/engine/defaults.py", line 176, in __call__
    predictions = self.model([inputs])[0]
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/app/detectron2/detectron2/modeling/meta_arch/panoptic_fpn.py", line 98, in forward
    images, features, proposals, gt_instances
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/app/detectron2/detectron2/modeling/roi_heads/cascade_rcnn.py", line 97, in forward
    pred_instances = self._forward_box(features_list, proposals)
  File "/app/detectron2/detectron2/modeling/roi_heads/cascade_rcnn.py", line 112, in _forward_box
    head_outputs.append(self._run_stage(features, proposals, k))
  File "/app/detectron2/detectron2/modeling/roi_heads/cascade_rcnn.py", line 203, in _run_stage
    box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/app/detectron2/detectron2/modeling/poolers.py", line 192, in forward
    output[inds] = pooler(x_level, pooler_fmt_boxes_level)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/app/detectron2/detectron2/layers/roi_align.py", line 95, in forward
    input, rois, self.output_size, self.spatial_scale, self.sampling_ratio, self.aligned
  File "/app/detectron2/detectron2/layers/roi_align.py", line 20, in forward
    input, roi, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned
RuntimeError: CUDA error: invalid device function (ROIAlign_forward_cuda at /app/detectron2/detectron2/layers/csrc/ROIAlign/ROIAlign_cuda.cu:359)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x47 (0x7ffa5c402687 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: ROIAlign_forward_cuda(at::Tensor const&, at::Tensor const&, float, int, int, int, bool) + 0xa37 (0x7ffa0653b6f5 in /app/detectron2/detectron2/_C.cpython-36m-x86_64-linux-gnu.so)
frame #2: ROIAlign_forward(at::Tensor const&, at::Tensor const&, float, int, int, int, bool) + 0xbc (0x7ffa064c9fdc in /app/detectron2/detectron2/_C.cpython-36m-x86_64-linux-gnu.so)
frame #3: <unknown function> + 0x5961a (0x7ffa064db61a in /app/detectron2/detectron2/_C.cpython-36m-x86_64-linux-gnu.so)
frame #4: <unknown function> + 0x5971e (0x7ffa064db71e in /app/detectron2/detectron2/_C.cpython-36m-x86_64-linux-gnu.so)
frame #5: <unknown function> + 0x53ca0 (0x7ffa064d5ca0 in /app/detectron2/detectron2/_C.cpython-36m-x86_64-linux-gnu.so)
<omitting python frames>
frame #12: THPFunction_apply(_object*, _object*) + 0x9ff (0x7ffa5d63dacf in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)

Environment

---------------------  -------------------------------------------------------------------
Python                 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) [GCC 7.3.0]
Detectron2 Compiler    GCC 5.4
DETECTRON2_ENV_MODULE  <not set>
PyTorch                1.3.0
PyTorch Debug Build    False
CUDA available         True
GPU 0,1                GeForce GTX 1080 Ti
Pillow                 6.2.0
cv2                    3.4.4
---------------------  -------------------------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v0.20.5 (Git Hash 0125f28c61c1f822fd48570b4c1066f96fcb9b2e)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CUDA Runtime 10.1
  - NVCC architecture flags: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_50,code=compute_50
  - CuDNN 7.6.3
  - Magma 2.5.1
  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=True, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 4
Comments: 21 (9 by maintainers)

Most upvoted comments

I was able to reproduce the same error when I use the wrong version of cuda.

What I did: I install pytorch from conda install pytorch torchvision cudatoolkit=10.1 -c pytorch, however my local cuda runtime and nvcc are in 10.0. In this case, I can observe the same error. Please check whether your cuda version is correct.

+30

ppwwyyxx on Oct 15, 2019

It seems that mismatched NVCC vs CUDA Runtime version is the root cause. Closing but feel free to reopen if this does not solve your issue.

ppwwyyxx on Oct 29, 2019

It seems that mismatched NVCC vs CUDA Runtime version is the root cause. Closing but feel free to reopen if this does not solve your issue.

Yes, that’s the key to solve my problem

Problem

first, briefly introduce my problem: I’m new to Detectron2 and only one GPU(GeForce GTX 1080Ti). I choose to build Detectron2 from Source:

# Or, to install it from a local clone:
git clone https://github.com/facebookresearch/detectron2.git
python -m pip install -e detectron2

everything is fine and detectron2 is installed successfully

$ python -m pip install -e detectron2
Obtaining file:///home/lab305/ZhuJian/detectron2
Requirement already satisfied: termcolor>=1.1 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from detectron2==0.1.3) (1.1.0)
Requirement already satisfied: Pillow in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from detectron2==0.1.3) (7.0.0)
Requirement already satisfied: yacs>=0.1.6 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from detectron2==0.1.3) (0.1.7)
Requirement already satisfied: tabulate in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from detectron2==0.1.3) (0.8.7)
Requirement already satisfied: cloudpickle in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from detectron2==0.1.3) (1.4.1)
Requirement already satisfied: matplotlib in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from detectron2==0.1.3) (3.1.2)
Requirement already satisfied: mock in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from detectron2==0.1.3) (4.0.2)
Requirement already satisfied: tqdm>4.29.0 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from detectron2==0.1.3) (4.46.0)
Requirement already satisfied: tensorboard in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from detectron2==0.1.3) (2.0.0)
Requirement already satisfied: fvcore>=0.1.1 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from detectron2==0.1.3) (0.1.1.post200513)
Requirement already satisfied: future in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from detectron2==0.1.3) (0.18.2)
Requirement already satisfied: pydot in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from detectron2==0.1.3) (1.4.1)
Requirement already satisfied: PyYAML in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from yacs>=0.1.6->detectron2==0.1.3) (5.3.1)
Requirement already satisfied: cycler>=0.10 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from matplotlib->detectron2==0.1.3) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from matplotlib->detectron2==0.1.3) (2.4.6)
Requirement already satisfied: numpy>=1.11 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from matplotlib->detectron2==0.1.3) (1.18.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from matplotlib->detectron2==0.1.3) (1.1.0)
Requirement already satisfied: python-dateutil>=2.1 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from matplotlib->detectron2==0.1.3) (2.8.1)
Requirement already satisfied: wheel>=0.26; python_version >= "3" in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from tensorboard->detectron2==0.1.3) (0.33.6)
Requirement already satisfied: protobuf>=3.6.0 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from tensorboard->detectron2==0.1.3) (3.11.2)
Requirement already satisfied: werkzeug>=0.11.15 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from tensorboard->detectron2==0.1.3) (0.16.0)
Requirement already satisfied: six>=1.10.0 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from tensorboard->detectron2==0.1.3) (1.13.0)
Requirement already satisfied: absl-py>=0.4 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from tensorboard->detectron2==0.1.3) (0.8.1)
Requirement already satisfied: setuptools>=41.0.0 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from tensorboard->detectron2==0.1.3) (44.0.0.post20200106)
Requirement already satisfied: markdown>=2.6.8 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from tensorboard->detectron2==0.1.3) (3.1.1)
Requirement already satisfied: grpcio>=1.6.3 in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from tensorboard->detectron2==0.1.3) (1.16.1)
Requirement already satisfied: portalocker in /home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages (from fvcore>=0.1.1->detectron2==0.1.3) (1.7.0)
Installing collected packages: detectron2
  Found existing installation: detectron2 0.1.3
    Uninstalling detectron2-0.1.3:
      Successfully uninstalled detectron2-0.1.3
  Running setup.py develop for detectron2
Successfully installed detectron2

but when I try to train

$ ./train_net.py   --config-file ../configs/PascalVOC-Detection/faster_rcnn_R_50_C4.yaml   --num-gpus 1 SOLVER.IMS_PER_BATCH 2 SOLVER.BASE_LR 0.0025
...
...
[06/14 17:31:31 d2.engine.train_loop]: Starting training from iteration 0
ERROR [06/14 17:31:32 d2.engine.train_loop]: Exception during training:
Traceback (most recent call last):
  File "/home/lab305/ZhuJian/detectron2/detectron2/engine/train_loop.py", line 132, in train
    self.run_step()
  File "/home/lab305/ZhuJian/detectron2/detectron2/engine/train_loop.py", line 215, in run_step
    loss_dict = self.model(data)
  File "/home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lab305/ZhuJian/detectron2/detectron2/modeling/meta_arch/rcnn.py", line 123, in forward
    _, detector_losses = self.roi_heads(images, features, proposals, gt_instances)
  File "/home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lab305/ZhuJian/detectron2/detectron2/modeling/roi_heads/roi_heads.py", line 426, in forward
    [features[f] for f in self.in_features], proposal_boxes
  File "/home/lab305/ZhuJian/detectron2/detectron2/modeling/roi_heads/roi_heads.py", line 410, in _shared_roi_transform
    x = self.pooler(features, boxes)
  File "/home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lab305/ZhuJian/detectron2/detectron2/modeling/poolers.py", line 214, in forward
    return self.level_poolers[0](x[0], pooler_fmt_boxes)
  File "/home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lab305/ZhuJian/detectron2/detectron2/layers/roi_align.py", line 95, in forward
    input, rois, self.output_size, self.spatial_scale, self.sampling_ratio, self.aligned
  File "/home/lab305/ZhuJian/detectron2/detectron2/layers/roi_align.py", line 20, in forward
    input, roi, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned
RuntimeError: CUDA error: invalid device function
[06/14 17:31:32 d2.engine.hooks]: Total training time: 0:00:00 (0:00:00 on hooks)
Traceback (most recent call last):
  File "./train_net.py", line 169, in <module>
    args=(args,),
  File "/home/lab305/ZhuJian/detectron2/detectron2/engine/launch.py", line 57, in launch
    main_func(*args)
  File "./train_net.py", line 157, in main
    return trainer.train()
  File "/home/lab305/ZhuJian/detectron2/detectron2/engine/defaults.py", line 402, in train
    super().train(self.start_iter, self.max_iter)
  File "/home/lab305/ZhuJian/detectron2/detectron2/engine/train_loop.py", line 132, in train
    self.run_step()
  File "/home/lab305/ZhuJian/detectron2/detectron2/engine/train_loop.py", line 215, in run_step
    loss_dict = self.model(data)
  File "/home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lab305/ZhuJian/detectron2/detectron2/modeling/meta_arch/rcnn.py", line 123, in forward
    _, detector_losses = self.roi_heads(images, features, proposals, gt_instances)
  File "/home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lab305/ZhuJian/detectron2/detectron2/modeling/roi_heads/roi_heads.py", line 426, in forward
    [features[f] for f in self.in_features], proposal_boxes
  File "/home/lab305/ZhuJian/detectron2/detectron2/modeling/roi_heads/roi_heads.py", line 410, in _shared_roi_transform
    x = self.pooler(features, boxes)
  File "/home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lab305/ZhuJian/detectron2/detectron2/modeling/poolers.py", line 214, in forward
    return self.level_poolers[0](x[0], pooler_fmt_boxes)
  File "/home/lab305/anaconda3/envs/pytorch1.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lab305/ZhuJian/detectron2/detectron2/layers/roi_align.py", line 95, in forward
    input, rois, self.output_size, self.spatial_scale, self.sampling_ratio, self.aligned
  File "/home/lab305/ZhuJian/detectron2/detectron2/layers/roi_align.py", line 20, in forward
    input, roi, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned
RuntimeError: CUDA error: invalid device function
段错误 (核心已转储)

Solve

I check the cuda version

# nvidia-smi
CUDA Version: 10.2
# nvcc --version
Cuda compilation tools, release 10.0, V10.0.130

before this I install cudatoolkit=10.2，but now i choose the earlier version

conda install pytorch torchvision cudatoolkit=10.0 -c pytorch

after rebuilt Detectron2，the problem solved！！！

zjZSTU on Jun 14, 2020

Most likely the solution to your problem is already in https://detectron2.readthedocs.io/tutorials/install.html#common-installation-issues. If you need help to solve an unexpected issue you observed, please include details following the issue template.

ppwwyyxx on Apr 1, 2020

No it’s unrelated to model zoo. It’s likely because cuda 9.2 is just what your computer is using.

ppwwyyxx on Oct 16, 2019

as followup from #78 . I installed new env with CUDA 9.2 and this solved my issue. Could the problem be since as stated at https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md all models are trained with CUDA 9.2 ?

batrlatom on Oct 16, 2019