unstructured: NLTK Error in partition_pdf(): Resource "punkt_tab" not found

Recent bug in partition_pdf():

2024-08-11T22:04:35.5170317Z ERROR    bbot.scanner:scanner.py:1195 Error in unstructured.handle_event(FILESYSTEM("{'path': '/tmp/.bbot_test/scans/testunstructured_test_5866686nbn/filedownload/20...", module=filedownload, tags={'file', 'filedownload', 'in-scope'})): /home/runner/work/bbot/bbot/bbot/modules/unstructured.py:100:handle_event(): 
2024-08-11T22:04:35.5170525Z **********************************************************************
2024-08-11T22:04:35.5170687Z   Resource punkt_tab not found.
2024-08-11T22:04:35.5170880Z   Please use the NLTK Downloader to obtain the resource:
2024-08-11T22:04:35.5170888Z 
2024-08-11T22:04:35.5171007Z   >>> import nltk
2024-08-11T22:04:35.5171150Z   >>> nltk.download('punkt_tab')
2024-08-11T22:04:35.5171247Z   
2024-08-11T22:04:35.5171440Z   For more information see: https://www.nltk.org/data.html
2024-08-11T22:04:35.5171454Z 
2024-08-11T22:04:35.5171681Z   Attempted to load tokenizers/punkt_tab/english/
2024-08-11T22:04:35.5171688Z 
2024-08-11T22:04:35.5171779Z   Searched in:
2024-08-11T22:04:35.5171916Z     - '/home/runner/nltk_data'
2024-08-11T22:04:35.5172236Z     - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/nltk_data'
2024-08-11T22:04:35.5172588Z     - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/share/nltk_data'
2024-08-11T22:04:35.5172916Z     - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/nltk_data'
2024-08-11T22:04:35.5173041Z     - '/usr/share/nltk_data'
2024-08-11T22:04:35.5173178Z     - '/usr/local/share/nltk_data'
2024-08-11T22:04:35.5173298Z     - '/usr/lib/nltk_data'
2024-08-11T22:04:35.5173427Z     - '/usr/local/lib/nltk_data'
2024-08-11T22:04:35.5173571Z **********************************************************************
2024-08-11T22:04:35.5173577Z 
2024-08-11T22:04:35.5173879Z TRACE    bbot.scanner:scanner.py:1196 concurrent.futures.process._RemoteTraceback: 
2024-08-11T22:04:35.5173965Z """
2024-08-11T22:04:35.5174075Z Traceback (most recent call last):
2024-08-11T22:04:35.5174605Z   File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
2024-08-11T22:04:35.5174792Z     r = call_item.fn(*call_item.args, **call_item.kwargs)
2024-08-11T22:04:35.5175188Z   File "/home/runner/work/bbot/bbot/bbot/modules/unstructured.py", line 152, in extract_text
2024-08-11T22:04:35.5175330Z     elements = partition(filename=file_path)
2024-08-11T22:04:35.5176078Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/auto.py", line 341, in partition
2024-08-11T22:04:35.5176192Z     elements = partition_pdf(
2024-08-11T22:04:35.5176968Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/documents/elements.py", line 605, in wrapper
2024-08-11T22:04:35.5177087Z     elements = func(*args, **kwargs)
2024-08-11T22:04:35.5177930Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/file_utils/filetype.py", line 706, in wrapper
2024-08-11T22:04:35.5178046Z     elements = func(*args, **kwargs)
2024-08-11T22:04:35.5178794Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/file_utils/filetype.py", line 662, in wrapper
2024-08-11T22:04:35.5178910Z     elements = func(*args, **kwargs)
2024-08-11T22:04:35.5179632Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/chunking/dispatch.py", line 74, in wrapper
2024-08-11T22:04:35.5179743Z     elements = func(*args, **kwargs)
2024-08-11T22:04:35.5180487Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/pdf.py", line 210, in partition_pdf
2024-08-11T22:04:35.5180664Z     return partition_pdf_or_image(
2024-08-11T22:04:35.5181472Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/pdf.py", line 357, in partition_pdf_or_image
2024-08-11T22:04:35.5181678Z     out_elements = _process_uncategorized_text_elements(elements)
2024-08-11T22:04:35.5182606Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/pdf.py", line 966, in _process_uncategorized_text_elements
2024-08-11T22:04:35.5182816Z     new_el = element_from_text(cast(Text, el).text)
2024-08-11T22:04:35.5183590Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/text.py", line 295, in element_from_text
2024-08-11T22:04:35.5183718Z     elif is_possible_narrative_text(text):
2024-08-11T22:04:35.5184568Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/text_type.py", line 80, in is_possible_narrative_text
2024-08-11T22:04:35.5184742Z     if exceeds_cap_ratio(text, threshold=cap_threshold):
2024-08-11T22:04:35.5185548Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/text_type.py", line 276, in exceeds_cap_ratio
2024-08-11T22:04:35.5185661Z     if sentence_count(text, 3) > 1:
2024-08-11T22:04:35.5186454Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/text_type.py", line 225, in sentence_count
2024-08-11T22:04:35.5186564Z     sentences = sent_tokenize(text)
2024-08-11T22:04:35.5187308Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/nlp/tokenize.py", line 137, in sent_tokenize
2024-08-11T22:04:35.5187414Z     return _sent_tokenize(text)
2024-08-11T22:04:35.5188138Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/nltk/tokenize/__init__.py", line 106, in sent_tokenize
2024-08-11T22:04:35.5188268Z     tokenizer = PunktTokenizer(language)
2024-08-11T22:04:35.5188964Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/nltk/tokenize/punkt.py", line 1744, in __init__
2024-08-11T22:04:35.5189063Z     self.load_lang(lang)
2024-08-11T22:04:35.5189762Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/nltk/tokenize/punkt.py", line 1749, in load_lang
2024-08-11T22:04:35.5189920Z     lang_dir = find(f"tokenizers/punkt_tab/{lang}/")
2024-08-11T22:04:35.5190537Z   File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/nltk/data.py", line 582, in find
2024-08-11T22:04:35.5190678Z     raise LookupError(resource_not_found)
2024-08-11T22:04:35.5190769Z LookupError: 
2024-08-11T22:04:35.5190911Z **********************************************************************
2024-08-11T22:04:35.5191070Z   Resource punkt_tab not found.
2024-08-11T22:04:35.5191326Z   Please use the NLTK Downloader to obtain the resource:
2024-08-11T22:04:35.5191332Z 
2024-08-11T22:04:35.5191452Z   >>> import nltk
2024-08-11T22:04:35.5191590Z   >>> nltk.download('punkt_tab')
2024-08-11T22:04:35.5191692Z   
2024-08-11T22:04:35.5191881Z   For more information see: https://www.nltk.org/data.html
2024-08-11T22:04:35.5191888Z 
2024-08-11T22:04:35.5192117Z   Attempted to load tokenizers/punkt_tab/english/
2024-08-11T22:04:35.5192122Z 
2024-08-11T22:04:35.5192210Z   Searched in:
2024-08-11T22:04:35.5192340Z     - '/home/runner/nltk_data'
2024-08-11T22:04:35.5192662Z     - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/nltk_data'
2024-08-11T22:04:35.5193010Z     - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/share/nltk_data'
2024-08-11T22:04:35.5193342Z     - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/nltk_data'
2024-08-11T22:04:35.5193467Z     - '/usr/share/nltk_data'
2024-08-11T22:04:35.5193660Z     - '/usr/local/share/nltk_data'
2024-08-11T22:04:35.5193791Z     - '/usr/lib/nltk_data'
2024-08-11T22:04:35.5193924Z     - '/usr/local/lib/nltk_data'
2024-08-11T22:04:35.5194067Z **********************************************************************
2024-08-11T22:04:35.5194072Z 
2024-08-11T22:04:35.5194150Z """
2024-08-11T22:04:35.5194265Z 
2024-08-11T22:04:35.5194508Z The above exception was the direct cause of the following exception:
2024-08-11T22:04:35.5194513Z 
2024-08-11T22:04:35.5194630Z Traceback (most recent call last):
2024-08-11T22:04:35.5194966Z   File "/home/runner/work/bbot/bbot/bbot/scanner/scanner.py", line 1172, in _acatch
2024-08-11T22:04:35.5195055Z     yield
2024-08-11T22:04:35.5195358Z   File "/home/runner/work/bbot/bbot/bbot/modules/base.py", line 637, in _worker
2024-08-11T22:04:35.5195472Z     await self.handle_event(event)
2024-08-11T22:04:35.5195851Z   File "/home/runner/work/bbot/bbot/bbot/modules/unstructured.py", line 100, in handle_event
2024-08-11T22:04:35.5196134Z     content = await self.scan.helpers.run_in_executor_mp(extract_text, file_path)
2024-08-11T22:04:35.5196227Z LookupError: 
2024-08-11T22:04:35.5196363Z **********************************************************************
2024-08-11T22:04:35.5196527Z   Resource punkt_tab not found.
2024-08-11T22:04:35.5196708Z   Please use the NLTK Downloader to obtain the resource:
2024-08-11T22:04:35.5196715Z 
2024-08-11T22:04:35.5196837Z   >>> import nltk
2024-08-11T22:04:35.5196970Z   >>> nltk.download('punkt_tab')
2024-08-11T22:04:35.5197064Z   
2024-08-11T22:04:35.5197260Z   For more information see: https://www.nltk.org/data.html
2024-08-11T22:04:35.5197265Z 
2024-08-11T22:04:35.5197488Z   Attempted to load tokenizers/punkt_tab/english/
2024-08-11T22:04:35.5197493Z 
2024-08-11T22:04:35.5197584Z   Searched in:
2024-08-11T22:04:35.5197710Z     - '/home/runner/nltk_data'
2024-08-11T22:04:35.5198025Z     - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/nltk_data'
2024-08-11T22:04:35.5198372Z     - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/share/nltk_data'
2024-08-11T22:04:35.5198700Z     - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/nltk_data'
2024-08-11T22:04:35.5198830Z     - '/usr/share/nltk_data'
2024-08-11T22:04:35.5198960Z     - '/usr/local/share/nltk_data'
2024-08-11T22:04:35.5199086Z     - '/usr/lib/nltk_data'
2024-08-11T22:04:35.5199218Z     - '/usr/local/lib/nltk_data'
2024-08-11T22:04:35.5199352Z **********************************************************************

Downstream issue: https://github.com/blacklanternsecurity/bbot/issues/1651

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 8
  • Comments: 35 (9 by maintainers)

Commits related to this issue

Most upvoted comments

Downgrading nltk to 3.8.1 seems to resolve the issue for me

New build unstructured 0.15.3 is available now in PyPI.

We’re working on a fix for this today.

Not sure who needs this, but I was able to fix this in the python interpreter:

python
import nltk
nltk.download('punkt_tab')

On the next run I got a similar error with another dependency and performed the same operation to download it.

Getting this error on 3.9.1 still.

Hi everyone - 0.15.5 is now available on PyPI. Appreciate the patience, there were a few issues that popped up this week that we were juggling simultaneously.

I also am wondering when the new build will be available - thank you

Thank you very much for the quick resolution of this problem! 👍 🙂

Question: How long do you think it will take until this version is available for PIP (https://pypi.org/project/unstructured/#history)?

I’m getting this error currently:

ERROR: No matching distribution found for unstructured==0.15.2

Thank you very much for the quick resolution of this problem! 👍 🙂

Question: How long do you think it will take until this version is available for PIP (https://pypi.org/project/unstructured/#history)?

I’m getting this error currently:

ERROR: No matching distribution found for unstructured==0.15.2

Again, thanks for the quick resolution! 🔥 I have the same question, my project is stopped waiting for the PIP, any update on that? Thanks in advance!

Everything was working yesterday, but as of today, I started encountering this error! Today’s setup included unstructured v.0.15.6 and nltk v.3.8.1 ( req’s from pip freeze attached).

Downgrading to unstructured v 0.15.5 resolved the issue.

0.15.6 is now available in PyPI.

Hi @oanamocean - try checking your nltk version. If it’s >=3.8.2, you’ll need to downgrade to 3.8.1. Also note that while there’s a CVE in 3.8.1, we don’t use the vulnerable code (which is nltk.download). We expect to remove that CVE from our dependency scans within the next week or two.

Merci beacoup @christinestraub

For onlookers, skip 0.15.3 and move to 0.15.4 as there is a bug. Although, as of Aug 14, 0.15.4 is not on PyPi…so…

Install for now with pip install "git+https://github.com/Unstructured-IO/unstructured@9b778e270dd8547476370a9417520679cd46c802#egg=unstructured[all-docs]" targeting the commit hash.

Same issue. Pinning 3.8.1 works.

For me the issue reemerged with unstructured==0.16.20. Version 0.16.19 still works

Hi @MuhammadBilal848

The new version unstructured==0.16.16 has been released. This version includes an enhancement where the necessary NLTK data will be automatically downloaded if the NLTK_DATA is missing during tokenization on import.

Thank you for your patience!

I tried numerous combinations of nltk and unstructured to try and resolve the punkt_tab issue, including unstructured>=0.15.7 and nltk>=3.9.0.

What ended up working for me was unstructured==0.15.5 and nltk==3.8.1.

UPDATE: After some more playing the following works (not sure how I missed this combo): unstructured==0.16.12 and nltk==3.9.1.

The latest version of unstructured (0.16.13) seems to break this again.

We had upgraded to 3.8.2, but that release was yanked from PyPI and we also observed a significant slowdown in document processing after upgrading nltk. There’s now a 3.9.1 release of nltk that looks like it will maintain support for the old model files and hopefully solve the speed issue. If it doesn’t, we’ll to replace nltk with another solution. We’ll have updates on this later in the week.

Merci beacoup @christinestraub For onlookers, skip 0.15.3 and move to 0.15.4 and there is a bug. Although, as of Aug 14, 0.15.4 is not on PyPi…so… Install for now with pip install "git+https://github.com/Unstructured-IO/unstructured@9b778e270dd8547476370a9417520679cd46c802#egg=unstructured[all-docs]" targeting the commit hash.

@isaackogan Thank you for the info! Could you share with us the bug that you have with the 0.15.3 version please?

There is a problem when installing 0.15.3: https://github.com/Unstructured-IO/unstructured/releases/tag/0.15.4

Resolve an installation error with pytesseract>=0.3.12 that occurred during pip install unstructured[pdf]==0.15.3.