datasets: Some languages in wikipedia dataset are not loading

Hi,

I am working with the wikipedia dataset and I have a script that goes over 92 of the available languages in that dataset. So far I have detected that ar, af, an are not loading. Other languages like fr and en are working fine. Here’s how I am loading them:

import nlp

langs = ['ar'. 'af', 'an']

for lang in langs:
    data = nlp.load_dataset('wikipedia', f'20200501.{lang}', beam_runner='DirectRunner', split='train') 
    print(lang, len(data))

Here’s what I see for ‘ar’ (it gets stuck there):

Downloading and preparing dataset wikipedia/20200501.ar (download: Unknown size, generated: Unknown size, post-processed: Unknown sizetotal: Unknown size) to /home/gaguilar/.cache/huggingface/datasets/wikipedia/20200501.ar/1.0.0/7be7f4324255faf70687be8692de57cf79197afdc33ff08d6a04ed602df32d50...

Note that those languages are indeed in the list of expected languages. Any suggestions on how to work around this? Thanks!

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 16 (11 by maintainers)

Most upvoted comments

Hey @gaguilar ,

I just found the “char2subword” paper and I’m really interested in trying it out on own vocabs/datasets like for historical texts (I’ve already trained some lms on newspaper articles with OCR errors).

Do you plan to release the code for your paper or is it possible to get the implementation 🤔 Many thanks 🤗

stefan-it on Oct 28, 2020

@lhoestq Any updates on this? I have similar issues with the Romanian dump, tnx.

D-Roberts on Oct 10, 2020

Ok, thanks for clarifying, that makes sense. I will time those examples later today and post back here.

Also, it seems that not all dumps should use the same date. For instance, I was checking the Spanish dump doing the following:

data = nlp.load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner', split='train')

I got the error below because this URL does not exist: https://dumps.wikimedia.org/eswiki/20200501/dumpstatus.json. So I checked the actual available dates here https://dumps.wikimedia.org/eswiki/ and there is no 20200501. If one tries for a date available in the link, then the nlp library does not allow such a request because is not in the list of expected datasets.

Downloading and preparing dataset wikipedia/20200501.es (download: Unknown size, generated: Unknown size, post-processed: Unknown sizetotal: Unknown size) to /home/gaguilar/.cache/huggingface/datasets/wikipedia/20200501.es/1.0.0/7be7f4324255faf70687be8692de57cf79197afdc33ff08d6a04ed602df32d50...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gaguilar/.conda/envs/pytorch/lib/python3.8/site-packages/nlp/load.py", line 548, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/gaguilar/.conda/envs/pytorch/lib/python3.8/site-packages/nlp/builder.py", line 462, in download_and_prepare
    self._download_and_prepare(
  File "/home/gaguilar/.conda/envs/pytorch/lib/python3.8/site-packages/nlp/builder.py", line 965, in _download_and_prepare
    super(BeamBasedBuilder, self)._download_and_prepare(
  File "/home/gaguilar/.conda/envs/pytorch/lib/python3.8/site-packages/nlp/builder.py", line 518, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/gaguilar/.conda/envs/pytorch/lib/python3.8/site-packages/nlp/datasets/wikipedia/7be7f4324255faf70687be8692de57cf79197afdc33ff08d6a04ed602df32d50/wikipedia.py", line 422, in _split_generators
    downloaded_files = dl_manager.download_and_extract({"info": info_url})
  File "/home/gaguilar/.conda/envs/pytorch/lib/python3.8/site-packages/nlp/utils/download_manager.py", line 220, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/gaguilar/.conda/envs/pytorch/lib/python3.8/site-packages/nlp/utils/download_manager.py", line 155, in download
    downloaded_path_or_paths = map_nested(
  File "/home/gaguilar/.conda/envs/pytorch/lib/python3.8/site-packages/nlp/utils/py_utils.py", line 163, in map_nested
    return {
  File "/home/gaguilar/.conda/envs/pytorch/lib/python3.8/site-packages/nlp/utils/py_utils.py", line 164, in <dictcomp>
    k: map_nested(
  File "/home/gaguilar/.conda/envs/pytorch/lib/python3.8/site-packages/nlp/utils/py_utils.py", line 191, in map_nested
    return function(data_struct)
  File "/home/gaguilar/.conda/envs/pytorch/lib/python3.8/site-packages/nlp/utils/download_manager.py", line 156, in <lambda>
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/home/gaguilar/.conda/envs/pytorch/lib/python3.8/site-packages/nlp/utils/file_utils.py", line 191, in cached_path
    output_path = get_from_cache(
  File "/home/gaguilar/.conda/envs/pytorch/lib/python3.8/site-packages/nlp/utils/file_utils.py", line 356, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://dumps.wikimedia.org/eswiki/20200501/dumpstatus.json

gaguilar on Sep 7, 2020

Hi ! This looks related to this issue: https://github.com/huggingface/datasets/issues/1994 Basically the parser that is used (mwparserfromhell) has some issues for some pages in es. We already reported some issues for es on their repo at https://github.com/earwig/mwparserfromhell/issues/247 but it looks like there are still a few issues. Might be a good idea to open a new issue on the mwparserfromhell repo

lhoestq on Jul 15, 2021

Hi ! The link https://dumps.wikimedia.org/idwiki/20210501/dumpstatus.json seems to be working fine for me.

Regarding the time outs, it must come either from an issue on the wikimedia host side, or from your internet connection. Feel free to try again several times.

lhoestq on Jun 25, 2021

This is an issue on the mwparserfromhell side. You could try to update mwparserfromhell and see if it fixes the issue. If it doesn’t we’ll have to create an issue on their repo for them to fix it. But first let’s see if the latest version of mwparserfromhell does the job.

lhoestq on Oct 29, 2020

Thanks ! This will be very helpful.

About the date issue, I think it’s possible to use another date with

load_dataset("wikipedia", language="es", date="...", beam_runner="...")

However we’ve not processed wikipedia dumps for other dates than 20200501 (yet ?)

One more thing that is specific to 20200501.es: it was available once but the mwparserfromhell was not able to parse it for some reason, so we didn’t manage to get a processed version of 20200501.es (see #321 )

lhoestq on Sep 8, 2020