snowball: KeyError: "Stemming algorithm 'porter' not found" on version 3.0.0 of the Python snowballstemmer package
The following code doesn’t work after upgrading to 3.0.0:
import snowballstemmer
snowballstemmer.stemmer('porter')
with the following error:
File "/path/to/venv/lib/python3.13/site-packages/snowballstemmer/__init__.py", line 27, in stemmer
raise KeyError("Stemming algorithm '%s' not found" % lang)
KeyError: "Stemming algorithm 'porter' not found"
It worked fine with v2.2.0. Tested on Python 3.13 and Python 3.11.
I don’t use snowballstemmer directly, but it’s used by Sphinx’s search functionality: https://github.com/sphinx-doc/sphinx/blob/c4929d026c8d22ba229b39cfc2250a9eb1476282/sphinx/search/en.py#L215
Not sure if this is a compatibility issue between Sphinx’s usage of the library or a genuine bug in the v3.0.0 release.
Thanks!
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 43
- Comments: 21 (9 by maintainers)
Commits related to this issue
- fix(lint): pin snowballstemmer due to snowballstem/snowball#229 — committed to canonical/charmed-kubeflow-chisme by orfeas-k 2 years ago
- fix(lint): pin snowballstemmer due to snowballstem/snowball#229 Close #143 — committed to canonical/charmed-kubeflow-chisme by orfeas-k 2 years ago
- fix(lint): pin snowballstemmer due to snowballstem/snowball#229 (#144) Close #143 — committed to canonical/charmed-kubeflow-chisme by orfeas-k 2 years ago
- Pin snowballstemmer version Snowballstemmer v3.0 released a few hours ago is causing issue with Sphinx. https://github.com/snowballstem/snowball/issues/229 Pin it for now. — committed to equinor/res2df by alifbe 2 years ago
- Fixate snowballstemmer version to `2.2.0` There is an issue with the `3.0.0`, see https://github.com/snowballstem/snowball/issues/229 for reference. — committed to gardener/cc-utils by 8R0WNI3 2 years ago
- Pin snowballstemmer==2.2.0 (https://github.com/snowballstem/snowball/issues/229) — committed to chungmin99/pyroki by brentyi 2 years ago
- Pin snowballstemmer<3 Because we depend on Sphinx, which depends on it. See: - https://github.com/snowballstem/snowball/issues/229 - https://github.com/sphinx-doc/sphinx/issues/13533 — committed to neuronsimulator/nrn by JCGoran 2 years ago
- Pin snowballstemmer version Snowballstemmer v3.0 released a few hours ago is causing issue with Sphinx. https://github.com/snowballstem/snowball/issues/229 Pin it for now. — committed to equinor/res2df by alifbe 2 years ago
- python: Specify correct dependencies for `$(python_output_dir)/__init__.py` When `python/create_init.py` is called, the `*_stemmer.py` should already exist, because this script builds a list of langu... — committed to mitya57/snowball by mitya57 2 years ago
- [Docs] Fix broken CI (#7797) fixing ci - https://github.com/mlrun/mlrun/actions/runs/14905616200/job/41867155912 ``` File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/snowbal... — committed to mlrun/mlrun by moranbental 2 years ago
- [Docs] Fix broken CI [1.8.x] (#7799) fixing ci - https://github.com/mlrun/mlrun/actions/runs/14905616200/job/41867155912 ``` File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages... — committed to mlrun/mlrun by moranbental 2 years ago
- [Docs] Fix broken CI [1.9.x] (#7801) fixing ci - https://github.com/mlrun/mlrun/actions/runs/14905616200/job/41867155912 ``` File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages... — committed to mlrun/mlrun by moranbental 2 years ago
- python: Specify correct dependencies for `$(python_output_dir)/__init__.py` When `python/create_init.py` is called, the `*_stemmer.py` should already exist, because this script builds a list of langu... — committed to snowballstem/snowball by mitya57 2 years ago
- python: Specify correct dependencies for `$(python_output_dir)/__init__.py` When `python/create_init.py` is called, the `*_stemmer.py` should already exist, because this script builds a list of langu... — committed to snowballstem/snowball by mitya57 2 years ago
- Documentation hotfixes and improvements (#76) This PR contains fixes and improvements to quickly publish the docs on Read The Docs. ## Issue * Some links and cross-references did not work * Python d... — committed to canonical/charmed-etcd-operator by a-velasco 2 years ago
I am sorry for the breakage. I yanked version 3.0.0 for now, and will investigate and upload new version.
@laymonage: We didn’t inspect the codebase yet, but your evaluation sounds very likely. [^1]
Dear maintainers: Please yank this release from PyPI to resolve regressions on many downstream pipelines. 🙏
[^1]: We observed similar hiccups on corresponding transitions on a few other projects, where we haven’t been careful enough, and because regular CI workflows usually don’t validate against packaging flaws. 🍀
This is now fixed in version 3.0.0.1 on PyPI.
PR #231 contains a fix for the build system to prevent this from happening in the future.
Reminder to everyone, pinning is not needed now that it’s yanked. Also, you don’t need to pin, you could have just used
snowballstemmer!=3.0.0, which protects against the broken version, but won’t come back to bite you later when something starts requiring >=3.pydocstyle which uses snowballstemmer has similar issues. Those can also be reproduced by:
which raises the KeyError “Stemming algorithm ‘english’ not found”
Here it is before: https://inspector.pypi.io/project/snowballstemmer/2.2.0/packages/ed/dc/c02e01294f7265e63a7315fe086dd1df7dacb9f840a804da846b96d01b96/snowballstemmer-2.2.0-py2.py3-none-any.whl/snowballstemmer/__init__.py
And after: https://inspector.pypi.io/project/snowballstemmer/3.0.0/packages/2c/3a/e5d32789ea254f83b2a5116bd877b0b127a4f5bbf6eb04f248c9d55b79b0/snowballstemmer-3.0.0-py3-none-any.whl/snowballstemmer/__init__.py
The languages dict is completely empty.
It might be a good idea to add an assert in
python/create_init.pyto make sure the languages dict is not empty before writing the file.I see this when using Flake8’s pydocstyle plugin:
And then there’s this:
Hi. That Sphinx/RTD build is also tripping on this.
We think it could be advisable to just yank the release from PyPI if it’s causing significant downstream issues on the many Sphinx installations/builds out there. Otherwise, would Sphinx be able to cope in any other way?
cc @ojwb – we’ve had a report of this in Sphinx, please let us know if we need to change our use of the API. I don’t see anything obvious in NEWS about this, though.
A
The fixes were in the creation process, there aren’t any code changes. So I’m not sure if there’s a need for a 3.0.1, 3.0.0.1 is fine unless there are other issues. The only problem was running
make dist_libstemmer_pythonon a fresh checkout produced broken wheels, there was a requiredmakestep before it. I happened to run the required pre-step because I was just trying things, that’s why it worked for me. That’s protected now so future releases are safer.I would recommend trying to make a proper build system that works from scratch without any pre-processing, of course, like scikit-build-core + cmake or hatchling and it’s build scripts. But that’s a bigger task. 😉
FYI, with trusted publishing, there are security benefits like SigStore signing that occur when publishing from CI. You can also protect the release environment and require a certain number of maintainers to sign off on it before it releases, for example. See https://learn.scientific-python.org/development/guides/gha-pure/ for an example of setup. Many of the PyPA’s packages use it, for example. It’s a different set of security issues, but not having to keep an all-powerful upload token and having a digital trail to your CI are some of the benefits.
Thanks! 3.0.0.1 worked for me 👍🏼
Hi again. @mitya57: Thank you for yanking the release, and thanks for the excellent hint about using
snowballstemmer!=3.0.0and for quickly finding the root cause, @henryiii.We also see the same ‘english’ issue in https://github.com/ssec/sift/pull/448 https://github.com/ssec/sift/actions/runs/14903088149/job/41859433281?pr=448