unstructured: NLTK Error in partition_pdf(): Resource "punkt_tab" not found
Recent bug in partition_pdf():
2024-08-11T22:04:35.5170317Z ERROR bbot.scanner:scanner.py:1195 Error in unstructured.handle_event(FILESYSTEM("{'path': '/tmp/.bbot_test/scans/testunstructured_test_5866686nbn/filedownload/20...", module=filedownload, tags={'file', 'filedownload', 'in-scope'})): /home/runner/work/bbot/bbot/bbot/modules/unstructured.py:100:handle_event():
2024-08-11T22:04:35.5170525Z **********************************************************************
2024-08-11T22:04:35.5170687Z Resource [93mpunkt_tab[0m not found.
2024-08-11T22:04:35.5170880Z Please use the NLTK Downloader to obtain the resource:
2024-08-11T22:04:35.5170888Z
2024-08-11T22:04:35.5171007Z [31m>>> import nltk
2024-08-11T22:04:35.5171150Z >>> nltk.download('punkt_tab')
2024-08-11T22:04:35.5171247Z [0m
2024-08-11T22:04:35.5171440Z For more information see: https://www.nltk.org/data.html
2024-08-11T22:04:35.5171454Z
2024-08-11T22:04:35.5171681Z Attempted to load [93mtokenizers/punkt_tab/english/[0m
2024-08-11T22:04:35.5171688Z
2024-08-11T22:04:35.5171779Z Searched in:
2024-08-11T22:04:35.5171916Z - '/home/runner/nltk_data'
2024-08-11T22:04:35.5172236Z - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/nltk_data'
2024-08-11T22:04:35.5172588Z - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/share/nltk_data'
2024-08-11T22:04:35.5172916Z - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/nltk_data'
2024-08-11T22:04:35.5173041Z - '/usr/share/nltk_data'
2024-08-11T22:04:35.5173178Z - '/usr/local/share/nltk_data'
2024-08-11T22:04:35.5173298Z - '/usr/lib/nltk_data'
2024-08-11T22:04:35.5173427Z - '/usr/local/lib/nltk_data'
2024-08-11T22:04:35.5173571Z **********************************************************************
2024-08-11T22:04:35.5173577Z
2024-08-11T22:04:35.5173879Z TRACE bbot.scanner:scanner.py:1196 concurrent.futures.process._RemoteTraceback:
2024-08-11T22:04:35.5173965Z """
2024-08-11T22:04:35.5174075Z Traceback (most recent call last):
2024-08-11T22:04:35.5174605Z File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
2024-08-11T22:04:35.5174792Z r = call_item.fn(*call_item.args, **call_item.kwargs)
2024-08-11T22:04:35.5175188Z File "/home/runner/work/bbot/bbot/bbot/modules/unstructured.py", line 152, in extract_text
2024-08-11T22:04:35.5175330Z elements = partition(filename=file_path)
2024-08-11T22:04:35.5176078Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/auto.py", line 341, in partition
2024-08-11T22:04:35.5176192Z elements = partition_pdf(
2024-08-11T22:04:35.5176968Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/documents/elements.py", line 605, in wrapper
2024-08-11T22:04:35.5177087Z elements = func(*args, **kwargs)
2024-08-11T22:04:35.5177930Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/file_utils/filetype.py", line 706, in wrapper
2024-08-11T22:04:35.5178046Z elements = func(*args, **kwargs)
2024-08-11T22:04:35.5178794Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/file_utils/filetype.py", line 662, in wrapper
2024-08-11T22:04:35.5178910Z elements = func(*args, **kwargs)
2024-08-11T22:04:35.5179632Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/chunking/dispatch.py", line 74, in wrapper
2024-08-11T22:04:35.5179743Z elements = func(*args, **kwargs)
2024-08-11T22:04:35.5180487Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/pdf.py", line 210, in partition_pdf
2024-08-11T22:04:35.5180664Z return partition_pdf_or_image(
2024-08-11T22:04:35.5181472Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/pdf.py", line 357, in partition_pdf_or_image
2024-08-11T22:04:35.5181678Z out_elements = _process_uncategorized_text_elements(elements)
2024-08-11T22:04:35.5182606Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/pdf.py", line 966, in _process_uncategorized_text_elements
2024-08-11T22:04:35.5182816Z new_el = element_from_text(cast(Text, el).text)
2024-08-11T22:04:35.5183590Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/text.py", line 295, in element_from_text
2024-08-11T22:04:35.5183718Z elif is_possible_narrative_text(text):
2024-08-11T22:04:35.5184568Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/text_type.py", line 80, in is_possible_narrative_text
2024-08-11T22:04:35.5184742Z if exceeds_cap_ratio(text, threshold=cap_threshold):
2024-08-11T22:04:35.5185548Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/text_type.py", line 276, in exceeds_cap_ratio
2024-08-11T22:04:35.5185661Z if sentence_count(text, 3) > 1:
2024-08-11T22:04:35.5186454Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/partition/text_type.py", line 225, in sentence_count
2024-08-11T22:04:35.5186564Z sentences = sent_tokenize(text)
2024-08-11T22:04:35.5187308Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/unstructured/nlp/tokenize.py", line 137, in sent_tokenize
2024-08-11T22:04:35.5187414Z return _sent_tokenize(text)
2024-08-11T22:04:35.5188138Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/nltk/tokenize/__init__.py", line 106, in sent_tokenize
2024-08-11T22:04:35.5188268Z tokenizer = PunktTokenizer(language)
2024-08-11T22:04:35.5188964Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/nltk/tokenize/punkt.py", line 1744, in __init__
2024-08-11T22:04:35.5189063Z self.load_lang(lang)
2024-08-11T22:04:35.5189762Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/nltk/tokenize/punkt.py", line 1749, in load_lang
2024-08-11T22:04:35.5189920Z lang_dir = find(f"tokenizers/punkt_tab/{lang}/")
2024-08-11T22:04:35.5190537Z File "/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/python3.9/site-packages/nltk/data.py", line 582, in find
2024-08-11T22:04:35.5190678Z raise LookupError(resource_not_found)
2024-08-11T22:04:35.5190769Z LookupError:
2024-08-11T22:04:35.5190911Z **********************************************************************
2024-08-11T22:04:35.5191070Z Resource [93mpunkt_tab[0m not found.
2024-08-11T22:04:35.5191326Z Please use the NLTK Downloader to obtain the resource:
2024-08-11T22:04:35.5191332Z
2024-08-11T22:04:35.5191452Z [31m>>> import nltk
2024-08-11T22:04:35.5191590Z >>> nltk.download('punkt_tab')
2024-08-11T22:04:35.5191692Z [0m
2024-08-11T22:04:35.5191881Z For more information see: https://www.nltk.org/data.html
2024-08-11T22:04:35.5191888Z
2024-08-11T22:04:35.5192117Z Attempted to load [93mtokenizers/punkt_tab/english/[0m
2024-08-11T22:04:35.5192122Z
2024-08-11T22:04:35.5192210Z Searched in:
2024-08-11T22:04:35.5192340Z - '/home/runner/nltk_data'
2024-08-11T22:04:35.5192662Z - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/nltk_data'
2024-08-11T22:04:35.5193010Z - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/share/nltk_data'
2024-08-11T22:04:35.5193342Z - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/nltk_data'
2024-08-11T22:04:35.5193467Z - '/usr/share/nltk_data'
2024-08-11T22:04:35.5193660Z - '/usr/local/share/nltk_data'
2024-08-11T22:04:35.5193791Z - '/usr/lib/nltk_data'
2024-08-11T22:04:35.5193924Z - '/usr/local/lib/nltk_data'
2024-08-11T22:04:35.5194067Z **********************************************************************
2024-08-11T22:04:35.5194072Z
2024-08-11T22:04:35.5194150Z """
2024-08-11T22:04:35.5194265Z
2024-08-11T22:04:35.5194508Z The above exception was the direct cause of the following exception:
2024-08-11T22:04:35.5194513Z
2024-08-11T22:04:35.5194630Z Traceback (most recent call last):
2024-08-11T22:04:35.5194966Z File "/home/runner/work/bbot/bbot/bbot/scanner/scanner.py", line 1172, in _acatch
2024-08-11T22:04:35.5195055Z yield
2024-08-11T22:04:35.5195358Z File "/home/runner/work/bbot/bbot/bbot/modules/base.py", line 637, in _worker
2024-08-11T22:04:35.5195472Z await self.handle_event(event)
2024-08-11T22:04:35.5195851Z File "/home/runner/work/bbot/bbot/bbot/modules/unstructured.py", line 100, in handle_event
2024-08-11T22:04:35.5196134Z content = await self.scan.helpers.run_in_executor_mp(extract_text, file_path)
2024-08-11T22:04:35.5196227Z LookupError:
2024-08-11T22:04:35.5196363Z **********************************************************************
2024-08-11T22:04:35.5196527Z Resource [93mpunkt_tab[0m not found.
2024-08-11T22:04:35.5196708Z Please use the NLTK Downloader to obtain the resource:
2024-08-11T22:04:35.5196715Z
2024-08-11T22:04:35.5196837Z [31m>>> import nltk
2024-08-11T22:04:35.5196970Z >>> nltk.download('punkt_tab')
2024-08-11T22:04:35.5197064Z [0m
2024-08-11T22:04:35.5197260Z For more information see: https://www.nltk.org/data.html
2024-08-11T22:04:35.5197265Z
2024-08-11T22:04:35.5197488Z Attempted to load [93mtokenizers/punkt_tab/english/[0m
2024-08-11T22:04:35.5197493Z
2024-08-11T22:04:35.5197584Z Searched in:
2024-08-11T22:04:35.5197710Z - '/home/runner/nltk_data'
2024-08-11T22:04:35.5198025Z - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/nltk_data'
2024-08-11T22:04:35.5198372Z - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/share/nltk_data'
2024-08-11T22:04:35.5198700Z - '/home/runner/.cache/pypoetry/virtualenvs/bbot-pd-UZ8Fz-py3.9/lib/nltk_data'
2024-08-11T22:04:35.5198830Z - '/usr/share/nltk_data'
2024-08-11T22:04:35.5198960Z - '/usr/local/share/nltk_data'
2024-08-11T22:04:35.5199086Z - '/usr/lib/nltk_data'
2024-08-11T22:04:35.5199218Z - '/usr/local/lib/nltk_data'
2024-08-11T22:04:35.5199352Z **********************************************************************
Downstream issue: https://github.com/blacklanternsecurity/bbot/issues/1651
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 8
- Comments: 35 (9 by maintainers)
Downgrading
nltkto 3.8.1 seems to resolve the issue for meNew build unstructured
0.15.3is available now inPyPI.We’re working on a fix for this today.
Not sure who needs this, but I was able to fix this in the python interpreter:
On the next run I got a similar error with another dependency and performed the same operation to download it.
Getting this error on 3.9.1 still.
Hi everyone -
0.15.5is now available on PyPI. Appreciate the patience, there were a few issues that popped up this week that we were juggling simultaneously.I also am wondering when the new build will be available - thank you
Thank you very much for the quick resolution of this problem! 👍 🙂
Question: How long do you think it will take until this version is available for PIP (https://pypi.org/project/unstructured/#history)?
I’m getting this error currently:
Again, thanks for the quick resolution! 🔥 I have the same question, my project is stopped waiting for the PIP, any update on that? Thanks in advance!
Everything was working yesterday, but as of today, I started encountering this error! Today’s setup included unstructured v.0.15.6 and nltk v.3.8.1 ( req’s from pip freeze attached).
Downgrading to unstructured v 0.15.5 resolved the issue.
0.15.6is now available in PyPI.Hi @oanamocean - try checking your
nltkversion. If it’s>=3.8.2, you’ll need to downgrade to3.8.1. Also note that while there’s a CVE in3.8.1, we don’t use the vulnerable code (which isnltk.download). We expect to remove that CVE from our dependency scans within the next week or two.Merci beacoup @christinestraub
For onlookers, skip
0.15.3and move to0.15.4as there is a bug. Although, as of Aug 14,0.15.4is not on PyPi…so…Install for now with
pip install "git+https://github.com/Unstructured-IO/unstructured@9b778e270dd8547476370a9417520679cd46c802#egg=unstructured[all-docs]"targeting the commit hash.Same issue. Pinning 3.8.1 works.
For me the issue reemerged with unstructured==0.16.20. Version 0.16.19 still works
Hi @MuhammadBilal848
The new version unstructured==0.16.16 has been released. This version includes an enhancement where the necessary NLTK data will be automatically downloaded if the NLTK_DATA is missing during tokenization on import.
Thank you for your patience!
I tried numerous combinations of nltk and unstructured to try and resolve the punkt_tab issue, including unstructured>=0.15.7 and nltk>=3.9.0.
What ended up working for me was unstructured==0.15.5 and nltk==3.8.1.
UPDATE: After some more playing the following works (not sure how I missed this combo): unstructured==0.16.12 and nltk==3.9.1.
The latest version of unstructured (0.16.13) seems to break this again.
We had upgraded to
3.8.2, but that release was yanked from PyPI and we also observed a significant slowdown in document processing after upgradingnltk. There’s now a3.9.1release ofnltkthat looks like it will maintain support for the old model files and hopefully solve the speed issue. If it doesn’t, we’ll to replacenltkwith another solution. We’ll have updates on this later in the week.There is a problem when installing 0.15.3: https://github.com/Unstructured-IO/unstructured/releases/tag/0.15.4