optimum: Pix2struct to ONNX execution error

System Info

Working on google colab, and installing optimum with: 
!python -m pip install git+https://github.com/huggingface/optimum.git#egg=optimum[onnxruntime]

Who can help?

Hi @fxmarty ! First of all, thank you for your work on implementing the pix2struct conversion here. I’m trying to run the conversion following the commands of the main page, but I am having some problems…

I have done the installation of optimum from the repositories as explained before, and to run the transformation I have try the following commands:

!optimum-cli export onnx -m fxmarty/pix2struct-tiny-random --optimize O2 fxmarty/pix2struct-tiny-random_onnx
!optimum-cli export onnx -m google/pix2struct-docvqa-base --optimize O2 pix2struct-docvqa-base_onnx
!optimum-cli export onnx -m google/pix2struct-base --optimize O2 pix2struct-base_onnx

…

These are the execution logs:

2023-06-15 08:14:28.116454: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Framework not specified. Using pt to export to ONNX.
Automatic task detection to image-to-text-with-past.
Downloading (…)okenizer_config.json: 100% 2.45k/2.45k [00:00<00:00, 9.20MB/s]
Downloading spiece.model: 100% 851k/851k [00:00<00:00, 1.87MB/s]
Downloading (…)/main/tokenizer.json: 100% 3.27M/3.27M [00:00<00:00, 6.58MB/s]
Downloading (…)cial_tokens_map.json: 100% 2.20k/2.20k [00:00<00:00, 8.83MB/s]
Downloading (…)rocessor_config.json: 100% 250/250 [00:00<00:00, 1.05MB/s]
Using framework PyTorch: 2.0.1+cu118
/usr/local/lib/python3.10/dist-packages/transformers/models/pix2struct/modeling_pix2struct.py:221: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  scores = torch.max(scores, torch.tensor(torch.finfo(scores.dtype).min))
============= Diagnostic Run torch.onnx.export version 2.0.1+cu118 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Using framework PyTorch: 2.0.1+cu118
/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:832: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if causal_mask.shape[1] < attention_mask.shape[1]:
============= Diagnostic Run torch.onnx.export version 2.0.1+cu118 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Traceback (most recent call last):
  File "/usr/local/bin/optimum-cli", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/optimum/commands/optimum_cli.py", line 163, in main
    service.run()
  File "/usr/local/lib/python3.10/dist-packages/optimum/commands/export/onnx.py", line 219, in run
    main_export(
  File "/usr/local/lib/python3.10/dist-packages/optimum/exporters/onnx/__main__.py", line 309, in main_export
    _, onnx_outputs = export_models(
  File "/usr/local/lib/python3.10/dist-packages/optimum/exporters/onnx/convert.py", line 613, in export_models
    export(
  File "/usr/local/lib/python3.10/dist-packages/optimum/exporters/onnx/convert.py", line 709, in export
    export_output = export_pytorch(
  File "/usr/local/lib/python3.10/dist-packages/optimum/exporters/onnx/convert.py", line 442, in export_pytorch
    onnx_export(
  File "/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py", line 506, in export
    _export(
  File "/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py", line 1548, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py", line 1113, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
  File "/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py", line 989, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py", line 893, in _trace_and_get_graph_from_model
    trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
  File "/usr/local/lib/python3.10/dist-packages/torch/jit/_trace.py", line 1268, in _get_trace_graph
    outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/jit/_trace.py", line 127, in forward
    graph, out = torch._C._create_graph_by_tracing(
  File "/usr/local/lib/python3.10/dist-packages/torch/jit/_trace.py", line 118, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1488, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/optimum/exporters/onnx/model_patcher.py", line 129, in patched_forward
    outputs = self.orig_forward(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/pix2struct/modeling_pix2struct.py", line 1761, in forward
    encoder_last_hidden_state=encoder_outputs.last_hidden_state,
AttributeError: 'tuple' object has no attribute 'last_hidden_state'

Is this the correct way to make the conversion to ONNX? Can you help me with this problem?

Thank you so much in advance! 😃

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, …)
• My own task or dataset (give details below)

Reproduction

!python -m pip install git+https://github.com/huggingface/optimum.git#egg=optimum[onnxruntime]
!optimum-cli export onnx -m fxmarty/pix2struct-tiny-random --optimize O2 fxmarty/pix2struct-tiny-random_onnx

Expected behavior

Model in ONNX