tvm: TVMError: Check failed: it != type_definitions.end(): There is no definition of static_tensor_float32_*

When I convert a PyTorch Bert model to TVM, there is a problem blow

  File "tvm/python/tvm/relay/frontend/pytorch.py", line 2673, in from_pytorch
    default_dtype=default_dtype)
  File "/tvm/python/tvm/relay/frontend/pytorch.py", line 2580, in convert_operators
    relay_out = relay_op(inputs, _get_input_types(op_node, default_dtype=default_dtype))
  File "tvm/python/tvm/relay/frontend/pytorch.py", line 255, in _impl
    inferred_shape = _infer_shape(data)
  File "tvm/python/tvm/relay/frontend/common.py", line 486, in infer_shape
    out_type = infer_type(inputs, mod=mod)
  File "tvm/python/tvm/relay/frontend/common.py", line 465, in infer_type
    new_mod = IRModule.from_expr(node)
  File "tvm/python/tvm/ir/module.py", line 222, in from_expr
    return _ffi_api.Module_FromExpr(expr, funcs, defs)
  File "tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 225, in __call__
    raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
  [bt] (8) tvm/build/libtvm.so(tvm::relay::ExprVisitor::VisitExpr_(tvm::relay::CallNode const*)+0x13c) [0x7f21a6454bdc]
  [bt] (7) tvm/build/libtvm.so(tvm::relay::ExprVisitor::VisitExpr(tvm::RelayExpr const&)+0x71) [0x7f21a6457711]
  [bt] (6) tvm/build/libtvm.so(tvm::relay::ExprFunctor<void (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)+0x5b) [0x7f21a64153eb]
  [bt] (5) tvm/build/libtvm.so(tvm::relay::ExprVisitor::VisitExpr_(tvm::relay::CallNode const*)+0x13c) [0x7f21a6454bdc]
  [bt] (4) tvm/build/libtvm.so(tvm::relay::ExprVisitor::VisitExpr(tvm::RelayExpr const&)+0x71) [0x7f21a6457711]
  [bt] (3) tvm/build/libtvm.so(tvm::relay::ExprFunctor<void (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)+0x5b) [0x7f21a64153eb]
  [bt] (2) tvm/build/libtvm.so(tvm::relay::TypeVarEVisitor::VisitExpr_(tvm::ConstructorNode const*)+0x35) [0x7f21a6266505]
  [bt] (1) tvm/build/libtvm.so(tvm::IRModuleNode::LookupTypeDef(tvm::GlobalTypeVar const&) const+0x14a) [0x7f21a5c34a4a]
  [bt] (0) tvm/build/libtvm.so(+0xdc19e7) [0x7f21a5c329e7]
  File "tvm/src/ir/module.cc", line 299
TVMError: Check failed: it != type_definitions.end(): There is no definition of static_tensor_float32_1_4_768_t

Part of graph code list below:

  _234 = torch.transpose(torch.matmul(scores22, v10), 1, 2)
  h10 = torch.contiguous(_234, memory_format=0)
  _235 = ops.prim.NumToTensor(torch.size(h10, 0))
  _236 = ops.prim.NumToTensor(torch.size(h10, 1))
  input114 = torch.view(h10, [int(_235), int(_236), 768])
  output68 = torch.matmul(input114, torch.t(CONSTANTS.c143))
  input115 = torch.add_(output68, CONSTANTS.c150, alpha=1)
  _237 = torch.dropout(input115, 0.10000000000000001, False)
  input116 = torch.add(input111, _237, alpha=1)
  input117 = torch.layer_norm(input116, [768], CONSTANTS.c3, CONSTANTS.c4, 9.9999999999999998e-13, True)
  output69 = torch.matmul(input117, torch.t(CONSTANTS.c151))
  x94 = torch.add_(output69, CONSTANTS.c152, alpha=1)
  _238 = torch.mul(torch.pow(x94, 3), CONSTANTS.c17)
  _239 = torch.mul(torch.add(x94, _238, alpha=1), CONSTANTS.c18)
  _240 = torch.mul(x94, CONSTANTS.c19)
  _241 = torch.add(torch.tanh(_239), CONSTANTS.c20, alpha=1)
  input118 = torch.mul(_240, _241)
  output70 = torch.matmul(input118, torch.t(CONSTANTS.c153))
  input119 = torch.add_(output70, CONSTANTS.c154, alpha=1)
  _242 = torch.dropout(input119, 0.10000000000000001, False)
  input120 = torch.add(input117, _242, alpha=1)
  encoder_out = torch.layer_norm(input120, [768], CONSTANTS.c3, CONSTANTS.c4, 9.9999999999999998e-13, True)
  _243 = torch.slice(encoder_out, 0, 0, 9223372036854775807, 1)
  input121 = torch.select(_243, 1, 0)
  input122 = torch.addmm(CONSTANTS.c156, input121, torch.t(CONSTANTS.c155), beta=1, alpha=1)
  input123 = torch.tanh(input122)
  input124 = torch.dropout(input123, 0.10000000000000001, False)
  input125 = torch.addmm(CONSTANTS.c158, input124, torch.t(CONSTANTS.c157), beta=1, alpha=1)
  r = torch.softmax(input125, 1, None)
  return torch.to(r, 6, False, False)

It failed when converting slice OP.

Has anybody seen this question before？

Thanks

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 16 (16 by maintainers)

Most upvoted comments

I don’t know what to do about it, other than wait for PyTorch people to fix dtype saving. For quantized models, I added a workaround, but I don’t know if we can come up with a similar workaround (piggy back on to Relay’s type inference).

For reference, here is Torchscript IR with or without save/load:

without save/load

graph(%x : Long(4:5, 5:1)):
  %1 : Long() = prim::Constant[value={1}]() # test.py:8:0
  %2 : int = prim::Constant[value=1]() # test.py:8:0
  %3 : Long(4:5, 5:1) = aten::sub(%x, %1, %2) # test.py:8:0
  %4 : int = prim::Constant[value=4]() # test.py:8:0
  %5 : bool = prim::Constant[value=0]() # test.py:8:0
  %6 : bool = prim::Constant[value=0]() # test.py:8:0
  %7 : None = prim::Constant()
  %y.1 : Long(4:5, 5:1) = aten::to(%3, %4, %5, %6, %7) # test.py:8:0
  %9 : int = prim::Constant[value=6]() # test.py:9:0
  %10 : bool = prim::Constant[value=0]() # test.py:9:0
  %11 : bool = prim::Constant[value=0]() # test.py:9:0
  %12 : None = prim::Constant()
  %y : Float(4:5, 5:1) = aten::to(%y.1, %9, %10, %11, %12) # test.py:9:0
  %14 : int = prim::Constant[value=0]() # test.py:10:0
  %15 : int = prim::Constant[value=0]() # test.py:10:0
  %16 : int = prim::Constant[value=9223372036854775807]() # test.py:10:0
  %17 : int = prim::Constant[value=1]() # test.py:10:0
  %18 : Float(4:5, 5:1) = aten::slice(%y, %14, %15, %16, %17) # test.py:10:0
  %19 : int = prim::Constant[value=1]() # test.py:10:0
  %20 : int = prim::Constant[value=0]() # test.py:10:0
  %21 : Float(4:5) = aten::select(%18, %19, %20) # test.py:10:0
  return (%21)

with save/load

graph(%self : __torch__.PlaceholderModule,
      %x.1 : Tensor):
  %2 : None = prim::Constant() # :0:0
  %3 : bool = prim::Constant[value=0]() # test.py:8:0
  %4 : Tensor = prim::Constant[value={1}]() # test.py:8:0
  %5 : int = prim::Constant[value=1]() # test.py:8:0
  %6 : int = prim::Constant[value=4]() # test.py:8:0
  %7 : int = prim::Constant[value=6]() # test.py:9:0
  %8 : int = prim::Constant[value=0]() # test.py:10:0
  %9 : int = prim::Constant[value=9223372036854775807]() # test.py:10:0
  %10 : Tensor = aten::sub(%x.1, %4, %5) # test.py:8:0
  %y.1 : Tensor = aten::to(%10, %6, %3, %3, %2) # test.py:8:0
  %y0.1 : Tensor = aten::to(%y.1, %7, %3, %3, %2) # test.py:9:0
  %13 : Tensor = aten::slice(%y0.1, %8, %8, %9, %5) # test.py:10:0
  %14 : Tensor = aten::select(%13, %5, %8) # test.py:10:0
  return (%14)

masahi on Sep 2, 2020

We don’t accept input_types argument, so I can’t run your script. Note that if you save/load your jitted models, dtype will be lost. See https://github.com/pytorch/pytorch/issues/39690 if this is the same problem I raised there.

masahi on Aug 19, 2020