learn2learn: Error downloading mini-imagenet data
Somehow I get the following error:
Downloading: /home/miranda9/data/l2l_data/mini-imagenet-cache-test.pkl
Traceback (most recent call last):
File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/learn2learn/vision/datasets/mini_imagenet.py", line 102, in __init__
self.data = pickle.load(f)
_pickle.UnpicklingError: invalid load key, '<'.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/dataloaders/meta_learning/l2l_mini_imagenet_mi.py", line 268, in <module>
mi_test()
File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/dataloaders/meta_learning/l2l_mini_imagenet_mi.py", line 255, in mi_test
args.tasksets: BenchmarkTasksets = get_tasksets(
File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/dataloaders/meta_learning/l2l_mini_imagenet_mi.py", line 212, in get_tasksets
datasets, transforms = _TASKSETS[name](train_ways=train_ways,
File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/dataloaders/meta_learning/l2l_mini_imagenet_mi.py", line 78, in mini_imagenet_tasksets
test_dataset = l2l.vision.datasets.MiniImagenet(
File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/learn2learn/vision/datasets/mini_imagenet.py", line 108, in __init__
self.data = pickle.load(f)
_pickle.UnpicklingError: invalid load key, '<'.
but I only get it in the dgx machine not locally…has anyone had this error before?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 26 (4 by maintainers)
A temporal solution I found is the following:
1 Go to /learn2learn/vision/datasets and choose your <dataset>.py (for example, mini_imagenet.py). 2 Look through the code comments, there are the link from which the script download the test. 3 Click the links and download them with your web browser. 4 Move them to the folder the algorithm look for the dataset (by default ~/data).
This is how I have solved 😃
When I use the official code
train_dataset = l2l.vision.datasets.MiniImagenet(root='./data', mode='train'), I get the errorFileNotFoundError: [Errno 2] No such file or directory: './data/mini-imagenet-cache-train.pkl'. Currently, I can resolve this issue by manually downloading the file from Google Drive or Dropbox.However, upon checking the source code (https://github.com/learnables/learn2learn/blob/master/learn2learn/vision/datasets/mini_imagenet.py#L87), I noticed that the
download_pkl()function in the classMiniImagenet(data.Dataset)takes the following Google Drive file IDs as input:google_drive_file_id = '1wpmY-hmiJUUlRBkO9ZDCXAcIpHEFdOhD'google_drive_file_id = '1I3itTXpXxGV68olxM5roceUMG8itH9Xj'google_drive_file_id = '1KY5e491bkLFqJDp0-UWou3463Mo8AOco'. However, when I try to use gdown in Colab to download these files, I get the error"Access denied with the following error: Cannot retrieve the public link of the file. You may need to change the permission to 'Anyone with the link', or have had many accesses."Therefore, I believe that the issue may actually be with the permissions on Google Drive, which need to be set to “Anyone with the link” in order for thedownload_pkl()function to work properly.The current bug causing the error is that the function
download_file_from_google_drive()does not check for any error messages in theresponse, so the program assumes that the file has been successfully downloaded, when in fact nothing has been downloaded.Thanks for the notification @nightlessbaron.
I’ll re-open, and if the issue persists we can come up with a solution.
Just use gdown to download your data from google drive