TensorRT: ❓ [Question] Examples not working in nvcr.io/nvidia/pytorch:23.09-py3.

Thanks for the follow-up. It seems that in the newer transformers and diffusers versions, some of the operators have changed in the Stable Diffusion models, so our converter support is different which can affect performance. We have issues filed to support these as well, so the performance can be more uniform across transformers and diffusers versions.

Yes, I believe the Docker container should also work. You could alternatively also use the following for installing the latest nightly distribution of Torch-TRT:

pip install --pre torch torch-tensorrt --extra-index-url https://download.pytorch.org/whl/nightly/cu121

Hi @sayakpaul - thank you for the comments. Regarding the JIT workflow, this specific example was intended to illustrate how graph breaks are handled in torch.compile with the Torch-TRT backend. As you suggested, the model itself does not have sufficient operations to make the overhead of using a TRT engine worthwhile here. Additionally, this graph has an intentionally-added control-flow break, which means that a minimum of two TRT engines will be generated, further increasing the overhead for this small model. Since this model only has one computational operator per TRT block, the default options would not have converted these operators to TRT at all (“min_block_size”: 1 overrides this default). As a result, I believe the metrics are roughly as expected here.

The AOT workflow is not expected to work for this specific model (the one with the conditional). It would only work for the model without the conditional, as here. This is because the Torch ATen tracer cannot handle this sort of Python control flow. In order to use the AOT workflow on a model like this, the code would need to be modified to either remove the conditional or use tracer-allowed conditionals, like torch.cond

Hello - for the backend="tensorrt" issue, please ensure the line import torch_tensorrt is in the script (even though it may not be used directly). This is important because the import itself registers the backend, as here: https://github.com/pytorch/TensorRT/blob/0112bb4996d464f46ae8027f7d7146eb045d6364/py/torch_tensorrt/dynamo/backend/backends.py#L27 If this does not address the issue, could you please also share the version of Torch-TensorRT being used?

Regarding the AoT approach, @peri044 may be able to help further, but based on the error it sounds like the model may not be natively trace-able. Which specific example from the presentation are you referencing for the AoT sample?