poetry: poetry 1.2.0 can't install packages from private pypi servers supporting only MD5 hashes

FYI - I had success with working around this issue on Poetry 1.2 by disabling the experimental new installer

poetry config experimental.new-installer false

for those stuck on a repository without support but don’t want to downgrade

Since disabling new installer is now deprecated (since 1.4.0), previous workaround should be replaced by something more solid. Any updates on this issue?

<vent> I’m seriously being more and more angry with Poetry, last few months the tool is unusable and I’m seriously considering moving all projects to different package manager. I was waiting for 1.2.0 to solve all our issues with this tool, and while it did solve those issues, it created multiple new issues, which, again, prevents us to install our projects. I’m really sad, and I like Poetry a lot. </vent>

FYI - I had success with working around this issue on Poetry 1.2 by disabling the experimental new installer

poetry config experimental.new-installer false

for those stuck on a repository without support but don’t want to downgrade

If the new-installer is an experimental feature, please don’t enable by default 🙏

This issue doesn’t seem to be related only to private pypi servers supporting only MD5 hashes as I have a private artifactory repo that has md5, sha1, and sha256 hashes and poetry 1.2.0 is still refusing to install the packages. This was working before the 1.2.0 update as well.

It seems so far that this is only happening for internal private packages that are not published to pypi.

For anyone desperately looking for a solution. Hopefully this helps. I certainly got real frustrated by it.

On Macbook Pro M1 Pro MacOS: Monterey

1. Uninstall poetry 1.2.0 and install 1.1.5
2. Go to poetry’s own virtual env, in my case, ~/Library/Application Support/pypoetry/venv/bin
3. use ./pip list to check poetry-core’s version. I got 1.0.8
4. ./pip install poetry-core==1.0.4 to downgrade it

Downgrading poetry-core is due to it hardcodes sha256 when validating hashes. We’ll need to reverse that.

Well in that case we can close #4523 and leave this one open, because 1.1 is out of support so it’s only the 1.2 behaviour that’s worth tracking here.

Either way we only need one open issue.

For real, please stop tagging me: it was borderline rude the first time and it’s absurd after I’ve asked you to stop.

@shea-parkes Michael from @gemfury here. We support both SHA256 and MD5 checksums. Newer accounts default to SHA256, and we have recently added a switch to migrate older accounts. If you’re seeing MD5 checksums, we recommend heading to Settings/Features and migrating to SHA256.

Fair enough, going with this issue as the new canon.

I would like to point out that @dimbleby is NOT a Poetry maintainer – he is a frequent contributor and an expert in many of the hairier parts of the ecosystem and codebase. However, he has expressly declined to be declared formally part of the project, and should not be pinged on issues he has not explicitly expressed interest in.

I did find a stupid work around that seems to work for those using services like GemFury/Nexus that are still serving some packages with MD5 hashes only:

You can’t just replace the MD5 hash with the SHA256 hash in poetry.lock because Poetry still compares the reported hash from the repository to poetry.lock (and then goes on to hash the downloaded file with SHA256 and compare that against poetry.lock as well).

But at the moment, it appears you can have redundant file entries in poetry.lock. So instead of replacing the hash for a given .whl (or whatever file you’re failing on), duplicate that line and change the hash on the new line. So you have two lines describing the same file, but one has an md5 hash and one has a sha256 hash.

The poetry internals appear to currently build a set of all hashes for a given package, so it’s lenient on having multiple entries (as of current version of poetry).

For people suffering from this issue and using the latest (3.41.x or above) version of Sonatype Nexus for their private PyPi repository, the problem can be solved by forcing Nexus to re-create the index files for all affected packages. There is a Nexus bug where the problem is described: https://issues.sonatype.org/browse/NEXUS-34950 and the solution is to delete all files under simple/ in the Nexus PyPi server (or at least, all files for packages that aren’t yielding sha256 hashes yet). The files will be re-created properly next time a client requests package info via the simple interface.

I am facing the same issue here.

Another workaround could be to manually download the package(s) that causes the error from your private repo to your local machine. For example, you could put them in a sub-folder of your project and install it like this

poetry add ./<sub-folder>/problem_package.whl

… that @dimbleby …

omg now you’re at it…! I’m also not entirely sure about github tag etiquette and hope I don’t sound too grumpy; but would y’all please allow me to unsubscribe from this problem? It just happens not to be interesting to me, that’s all.

This isn’t duplicated @neersighted , I think @dimbleby also didn’t read the latest comments. 1.2.0 broke poetry for a lot of its current users. downgrading works.

FYI - I had success with working around this issue on Poetry 1.2 by disabling the experimental new installer

poetry config experimental.new-installer false

for those stuck on a repository without support but don’t want to downgrade

This worked for me but yeah it’s not recommended as @neersighted has mentioned.

The naming of the option as experimental.new-installer is kind of confusing though, because even I assumed (like @zen-xu) that the new installer is an experimental feature instead of disabling the new installer being an experimental option.

@invokermain you must upgrade Nexus to 3.41+ first, then you can:

• push new versions of the packages
• use that trick (remove simple/) to force sha256 hash creation

I am also facing the same problem and this is a major blocker in adopting poetry. Getting following error while installing a privately hosted library in nexus.

Retrieved digest for link mal_lib_poc-0.5.2.tar.gz(md5:0d08bdd0f8a21cb012816e77cec1d9c1) not in poetry.lock metadata ['sha256:605264c4a71648bbf4d966bc3031ac4121b1ecfea5ae730fe46257a36857233f', 'sha256:c021d5b2c34b68aeecb813a9108ddf3d9b19e2ebb1a93b4cb24a489c80417dd9']

  at ~/.local/share/pypoetry/venv/lib/python3.7/site-packages/poetry/installation/chooser.py:145 in _get_links
      141│             selected_links.append(link)
      142│ 
      143│         if links and not selected_links:
      144│             raise RuntimeError(
    → 145│                 f"Retrieved digest for link {link.filename}({h}) not in poetry.lock"
      146│                 f" metadata {hashes}"
      147│             )
      148│ 
      149│         return selected_links

stable except for the “edge case” that’s apparently burning everyone chiming in on this issue. What’s the status of the fix?

Well in that case we can close #4523 and leave this one open, because 1.1 is out of support so it’s only the 1.2 behaviour that’s worth tracking here.

Either way we only need one open issue.

For real, please stop tagging me: it was borderline rude the first time and it’s absurd after I’ve asked you to stop.

Apologies, you mentioned “don’t tag me in this issue”, and this is a separate issue from my eyes, thought you hadn’t read the comments above it in the other issue (as the thread changed context) I won’t be tagging you anymore regarding this. I don’t know the etiquette on tagging in GH, just saw you were a maintainer.

For pypi-server, start with flag --hash-algo=sha256 also fix the error.

pypi-server run . --hash-algo=sha256

I have a similar problem Any direction for a solution with poetry >= 1.2?

I can work around the issue by manually installing the offending package(s) via pip, but that doesn’t address the real issue here.

I’m not that deep into this topic. After reading several linked issues and PRs, I conclude as follows regarding the two options:

1. Respect the hash served by the server and do not recalculate.

#2958 introduced the change and probably did not allow md5 because it’s too weak (i.e. not secure). Further, this may result in other issues if the server is just a mirror, see https://github.com/python-poetry/poetry/pull/5326#issuecomment-1070927042:

Allowing non-sha checksums would break exporting to requirements.txt for use with pip (unless using --without-hashes) as pip no longer supports using md5, sha1, and sha224 checksums.

pip says:

However, weaker ones such as md5, sha1, and sha224 are excluded to avoid giving a false sense of security.

I would consider simply allowing md5 to be a security risk. I think a config option to allow md5 that is turned off by default and maybe still prints a warning if md5 is used when turned on might be a viable option.

2. Make the same dynamic calculation in chooser.

Sounds like a “simple” bugfix to me so I’d say this might make more sense. Further, it’s more secure. IMO, we should prefer this solution.

So the problem comes from https://github.com/python-poetry/poetry/blame/master/src/poetry/repositories/http_repository.py#L224-L246, change made by @abn; this ignores the hash returned by the HTTP repository and instead calculates SHA256 variant for it. However, at https://github.com/python-poetry/poetry/blob/master/src/poetry/installation/chooser.py#LL105C1-L105C1 this transformation is not made, hence the failure here. Options:

1. Respect the hash served by the server and do not recalculate.
2. Make the same dynamic calculation in chooser.

Maintainers, should I make a PR, any preferences? For now created a PR with the later.

I´m using Nexus v3.37, but how this is related to Nexus ? @ademoverflow

Because Nexus prior to 3.41 doesn’t report sha256 hashes, it only reports md5. AFAICT, Poetry 1.2+ does not support md5 hashes. You should upgrade to Nexus 3.44 if you need poetry 1.2+ or stick to poetry 1.1.x.

The new installer has been stable for several years, and disabling it is the experimental option. Doing so is under-tested and deviates from documented behavior.

FWIW, we ran into an issue that after upgrading Nexus, private packages still had the md5 checksum: Retrieved digest for link <package_name>-<ver>-py3-none-any.whl(md5:...) not in poetry.lock metadata ['sha256:...'] from API response of <nexus_url>/repository/<repository_name>/simple/<internal_package_name> (each href in the response text includes #md5:...)

This is a known issue for packages uploaded pre-Nexus upgrade: https://issues.sonatype.org/browse/NEXUS-34950 and the only resolution seems to be to re-upload the files (curl, delete, publish) or release new incremental versions

@shea-parkes Michael from @gemfury here. We support both SHA256 and MD5 checksums. Newer accounts default to SHA256, and we have recently added a switch to migrate older accounts. If you’re seeing MD5 checksums, we recommend heading to Settings/Features and migrating to SHA256.

Thanks so much for posting this here… I would have never found it… our CI was inexplicably broken for 3 hours today and I was going crazy reading these threads … 🍻

Some of you using pypiserver might find changing the hash algo on your server to sha256 as a solution (it did get lost a bit in the hidden comments in the related issue):

https://github.com/python-poetry/poetry/issues/4523#issuecomment-923741164

For people who are using pypiserver you can update the --hash-algo without any side effects (at least not on my side).

I can confirm that this works for us too.

Sounds good, apologies again!

Getting back in this subject, from @msscaroso on the #4523 issue

Same issue happening here. I’m using poetry-1.2.0

Poetry create the lock file with md5 and now it only looks for sha256. The package repository has both and the install should not fail.

I’ll try to pin poetry version and see if it works for the time being

I wonder if this was the regression ? It seems like in the case above the repo has both md5 and sha256, but poetry is not looking for the sha256 one. Anyho, will get back to work. For now I’m pinned to the latest version of 1.1.x and it “fixed” this problem.

Bumping this issue. Is there any resolution to this problem yet?

I’m not a maintainer so can’t say.

Just for clarification, when updating Nexus to 3.44+ it’s not necessary to reupload all packages. There are maintenance tasks that you can run to clear and generate the simple index https://issues.sonatype.org/browse/NEXUS-36608.

@ademoverflow

After upgrading to 3.46 and reuploading my packages, it works perfectly. Thanks !

I don’t want to take this too far off topic, but was reuploading necessary? Nexus documentation via https://issues.sonatype.org/browse/NEXUS-34950 implies this isn’t necessary after 3.44 😕

I’ve tested before reuploading with no success unfortunately. But right after the reupload, poetry succeed to add the package without any error.

@ademoverflow

After upgrading to 3.46 and reuploading my packages, it works perfectly. Thanks !

I don’t want to take this too far off topic, but was reuploading necessary? Nexus documentation via https://issues.sonatype.org/browse/NEXUS-34950 implies this isn’t necessary after 3.44 😕

On the plus side, the number of repositories that return md5 hashes is ever-dwindling

• artifactory https://www.jfrog.com/jira/browse/RTFACT-18495
• nexus https://issues.sonatype.org/browse/NEXUS-24127
• gemfury per a comment in this thread https://github.com/python-poetry/poetry/issues/6301#issuecomment-1241416832

I understand that in some organisations it takes a while to update these things, and probably there are others that haven’t yet caught up.

So while I’m not exactly advocating for “do nothing” here: it is true that as time passes, that becomes a more reasonable choice.

I thought there might have been a way to upload and replace, but we found it easier to release patch versions because we prevent replacing releases

What are the implications of making this switch? I assume we would have to update our poetry.lock files.

Yes, you’ll need to update the checksums in your poetry.lock, otherwise you’re risking failed installations.

@shea-parkes Michael from @gemfury here. We support both SHA256 and MD5 checksums. Newer accounts default to SHA256, and we have recently added a switch to migrate older accounts. If you’re seeing MD5 checksums, we recommend heading to Settings/Features and migrating to SHA256.

Thank you very much sir. I went to the account settings and found that switch. It worked nicely.

I had opened a Gemfury support ticket on Sept 6th asking about switching to all SHA256 hashes (sorry I didn’t find the self-service migration switch). I just checked my inbox just now and still don’t see a response. Would you be able to nudge the support department to respond and close that one down? (My Gemfury account is shea-parkes as well).

Same problem here! Temporary solution is pin version to <1.2