vision: RuntimeError: No such operator torchvision::nms

Similar to #1405, but I’m seeing this issue in the nightly builds.

What I did:

I download two nightly wheels from https://download.pytorch.org/whl/nightly/cu100/torch_nightly.html, and installed them with:

pip install torch-1.4.0.dev20191017+cu100-cp37-cp37m-linux_x86_64.whl torchvision-0.5.0.dev20191017+cu100-cp37-cp37m-linux_x86_64.whl

Then, I recompile detectron2; Running a simple detectron2 evaluation raises this error:

python tools/train_net.py --config-file configs/LVIS-InstanceSegmentation/mask_rcnn_R_101_FPN_1x.yaml --num-gpus 2 --eval-only DATALOADER.NUM_WORKERS 1 MODEL.WEIGHTS ~/data/D2models/LVIS-InstanceSegmentation/mask_rcnn_R_101_FPN_1x/144219035/model_final_824ab5.pkl

an example stack trace:

Traceback (most recent call last):
  File "tools/train_net.py", line 162, in <module>
    args=(args,),
  File "/private/home/yuxinwu/detectron2/detectron2/engine/launch.py", line 52, in launch
    main_func(*args)
  File "tools/train_net.py", line 133, in main
    res = Trainer.test(cfg, model)
  File "/private/home/yuxinwu/detectron2/detectron2/engine/defaults.py", line 413, in test
    results_i = inference_on_dataset(model, data_loader, evaluator)
  File "/private/home/yuxinwu/detectron2/detectron2/evaluation/evaluator.py", line 118, in inference_on_dataset
    outputs = model(inputs)
  File "/private/home/yuxinwu/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/private/home/yuxinwu/detectron2/detectron2/modeling/meta_arch/rcnn.py", line 66, in forward
    return self.inference(batched_inputs)
  File "/private/home/yuxinwu/detectron2/detectron2/modeling/meta_arch/rcnn.py", line 119, in inference
    proposals, _ = self.proposal_generator(images, features, None)
  File "/private/home/yuxinwu/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/private/home/yuxinwu/detectron2/detectron2/modeling/proposal_generator/rpn.py", line 179, in forward
    self.training,
  File "/private/home/yuxinwu/detectron2/detectron2/modeling/proposal_generator/rpn_outputs.py", line 136, in find_top_rpn_proposals
    keep = batched_nms(boxes.tensor, scores_per_img, lvl, nms_thresh)
  File "/private/home/yuxinwu/detectron2/detectron2/layers/nms.py", line 17, in batched_nms
    return box_ops.batched_nms(boxes, scores, idxs, iou_threshold)
  File "/private/home/yuxinwu/miniconda3/lib/python3.7/site-packages/torchvision/ops/boxes.py", line 70, in batched_nms
    keep = nms(boxes_for_nms, scores, iou_threshold)
  File "/private/home/yuxinwu/miniconda3/lib/python3.7/site-packages/torchvision/ops/boxes.py", line 31, in nms
    return torch.ops.torchvision.nms(boxes, scores, iou_threshold)
  File "/private/home/yuxinwu/miniconda3/lib/python3.7/site-packages/torch/_ops.py", line 61, in __getattr__
    op = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator torchvision::nms

another stack trace: https://github.com/facebookresearch/detectron2/issues/94#issue-507812638

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 4
Comments: 36 (16 by maintainers)

Commits related to this issue

bumping torch/torchvision to resolve: https://github.com/pytorch/vision/issues/1489 — committed to vdabravolski/detectron2-sagemaker by deleted user 4 years ago

Most upvoted comments

Ok, I could reproduce the problem thanks to @pedrofreire snippet. It seems to be due to installing the package with python setup.py install, while I’ve always used python setup.py build develop in order to be able to modify the library without reinstalling it.

I’m looking into a fix

fmassa on Oct 30, 2019

I get the similar error. And fixed it by removing the torchvision that I install by pip. I think it should be the version mismatch (I build torch from source)

pyjhzwh on Jun 22, 2022

Not right now (for OSS builds). We’ll look into it, but it is much easier to get a small app size if you’re able to bring all of the code into a single library.

dreiss on Nov 15, 2019

tried it, did not work. on the other hand if i do setup.py build develop it works just fine, so you are probably right, i just did not clean out something, maybe cache…

tangorn on Oct 31, 2019

@tangorn so your torchvision install might be picking the wrong _C.so file.

I’d recommend removing the try: in https://github.com/pytorch/vision/blob/13b35ffaa5167f3713ea7a53c43395d90b3a7cbc/torchvision/extension.py#L15-L19 so that you can further debug the issue. It’s probably related to the fact that in your setup https://github.com/pytorch/vision/blob/13b35ffaa5167f3713ea7a53c43395d90b3a7cbc/torchvision/extension.py#L10-L11 is not picking the right library

fmassa on Oct 29, 2019

Thanks for the detailed report!

I think the issue is that since https://github.com/pytorch/vision/pull/1467 we now build torchvision nightlies against PyTorch 1.3, and not PyTorch master.

The reason why I changed it to PyTorch 1.3 was because I was planning on cutting a new release of torchvision in the next few days, so it should be cut against PyTorch 1.3 and we needed to test that CI is working with PyTorch 1.3

But I think there is no need that the nightlies to be built based on PyTorch 1.3

fmassa on Oct 18, 2019