datasets: Datasets created with `push_to_hub` can't be accessed in offline mode

Hi ! Yes you can save your dataset locally with my_dataset.save_to_disk("path/to/local") and reload it later with load_from_disk("path/to/local")

(removing myself from assignees since I’m currently not working on this right now)

I can confirm that this problem is still happening with datasets 2.17.0, installed from pip

Hi, I’m having the same issue. Is there any update on this?

It seems that this does not support dataset with uppercase name

I started a PR to draft the logic to reload datasets from the cache fi they were created with push_to_hub: https://github.com/huggingface/datasets/pull/6459

Feel free to try it out

Any idea @lhoestq who to tag to fix this ? This is a very annoying bug, which is becoming more and more present since the push_to_hub API is getting used more ?

importable_directory_path is used to find a dataset script that was previously downloaded and cached from the Hub

However in your case there’s no dataset script on the Hub, only parquet files. So the logic must be extended for this case.

In particular I think you can add a new logic in the case where hashes is None (i.e. if there’s no dataset script associated to the dataset in the cache).

In this case you can check directly in the in the datasets cache for a directory named <namespace>__parquet and a subdirectory named <config_id>. The config_id must match {self.name.replace("/", "--")}-*.

In your case those two directories correspond to allenai___parquet and then allenai--my_dataset-def9ee5552a1043e

Then you can find the most recent version of the dataset in subdirectories (e.g. sorting using the last modified time of the dataset_info.json file).

Finally, we will need return the module that is used to load the dataset from the cache. It is the same module than the one that would have been normally used if you had an internet connection.

At that point you can ping me, because we will need to pass all this:

• module_path = _PACKAGED_DATASETS_MODULES["parquet"][0]
• hash it corresponds the name of the directory that contains the .arrow file, inside <namespace>__parquet/<config_id>
• builder_kwargs = {"hash": hash, "repo_id": self.name, "config_id": config_id} and currently config_id is not a valid argument for a DatasetBuilder

I think in the future we want to change this caching logic completely, since I don’t find it super easy to play with.