pytorch-forecasting: [BUG] Temporal Fusion Transformer trained on GPU then loaded on CPU does not propagate map_location into the loss metrics as expected
Describe the bug I have trained a TFT on a machine with several GPU’s. When loading the model on a smaller CPU-only machine as with:
model = TemporalFusionTransformer.load_from_checkpoint(model_file, map_location=torch.device('cpu'))
I get an error thrown:
AssertionError: Torch not compiled with CUDA enabled
I have tracked the problem to the loss metric for the model (QuantileLoss in my case) being initialised with a cuda device rather than the cpu device implied from the map_locations keyword arg above.
Here the full trace:
Traceback (most recent call last):
File "/Users/aaronrizzuto/code/spot_price_forecast/forecast_reals_model/model_evaluation.py", line 45, in <module>
main(model_path)
File "/Users/aaronrizzuto/code/spot_price_forecast/forecast_reals_model/model_evaluation.py", line 22, in main
model = TemporalFusionTransformer.load_from_checkpoint(model_file, map_location='cpu')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aaronrizzuto/code/spot_price_forecast/forecast_reals_model/venv/lib/python3.12/site-packages/lightning/pytorch/utilities/model_helpers.py", line 125, in wrapper
return self.method(cls, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aaronrizzuto/code/spot_price_forecast/forecast_reals_model/venv/lib/python3.12/site-packages/lightning/pytorch/core/module.py", line 1581, in load_from_checkpoint
loaded = _load_from_checkpoint(
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aaronrizzuto/code/spot_price_forecast/forecast_reals_model/venv/lib/python3.12/site-packages/lightning/pytorch/core/saving.py", line 100, in _load_from_checkpoint
return model.to(device)
^^^^^^^^^^^^^^^^
File "/Users/aaronrizzuto/code/spot_price_forecast/forecast_reals_model/venv/lib/python3.12/site-packages/lightning/fabric/utilities/device_dtype_mixin.py", line 55, in to
return super().to(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aaronrizzuto/code/spot_price_forecast/forecast_reals_model/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1340, in to
return self._apply(convert)
^^^^^^^^^^^^^^^^^^^^
File "/Users/aaronrizzuto/code/spot_price_forecast/forecast_reals_model/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 900, in _apply
module._apply(fn)
File "/Users/aaronrizzuto/code/spot_price_forecast/forecast_reals_model/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 900, in _apply
module._apply(fn)
File "/Users/aaronrizzuto/code/spot_price_forecast/forecast_reals_model/venv/lib/python3.12/site-packages/torchmetrics/metric.py", line 908, in _apply
_dummy_tensor = fn(torch.zeros(1, device=self.device))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aaronrizzuto/code/spot_price_forecast/forecast_reals_model/venv/lib/python3.12/site-packages/torch/cuda/__init__.py", line 310, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
is there a way to propagate the location_mapping into the loss metrics for the model?
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 15
Here is another solution.
In this case you initialise the TFT model from the inference feature dataset, then you replace the internals of the model with the trained model parameters, removing the
lossandlogging_metricssince these are the components that retain theircudadependency no matter what you do.This is the same for any TFT, I would recommend looking at some of the great example uses that the pytorch-forecasting devs have created.
E.g. https://pytorch-forecasting.readthedocs.io/en/latest/tutorials/stallion.html
Hi @saurabh-sh704 , here’s the block that does everything, creation of model and trainer are abstracted away here. Also, I’m not using bestTFT, I have a fixed process for training with parameters tuned with early stopping set up appropriately for my use case. The idea here is to load the model on a machine with gpu cuda, put it all onto the cpu, then save it.