poetry: Cannot download packages from private PyPi repository using HTTP basic auth with Poetry 1.1.0 + old v1.0.x PypiCloud w/ default settings

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: Fedora 32
Poetry version: Poetry 1.1.0
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/MasterNayru/0014ec89ae05d8f7651d18d8a107ec90

Issue

We have been using Poetry to pull down packages from a private PyPi repository and everything has been working fine until Poetry 1.1.0. We are configuring poetry to talk to our private PyPi installation by HTTP basic auth, and that auth works perfectly fine to resolve which versions of a package to install. The problem seems to be that that same auth is then used in the requests to download wheels from PyPi, which causes the following error to occur:

$ poetry config http-basic.myprivaterepo <username> <password>
$ poetry update -vvv

<snip>

   2  ~/.pyenv/versions/3.8.5/lib/python3.8/site-packages/poetry/repositories/pypi_repository.py:454 in _download
       452│ 
       453│     def _download(self, url, dest):  # type: (str, str) -> None
     → 454│         return download_file(url, dest, session=self.session)
       455│ 
       456│     def _log(self, msg, level="info"):

   1  ~/.pyenv/versions/3.8.5/lib/python3.8/site-packages/poetry/utils/helpers.py:98 in download_file
        96│ 
        97│     with get(url, stream=True) as response:
     →  98│         response.raise_for_status()
        99│ 
       100│         with open(dest, "wb") as f:

  HTTPError

  400 Client Error: Bad Request for url: https://deckard-pip.s3.amazonaws.com/1234/my_broken_dependency/my_broken_dependency-0.1.3-py3-none-any.whl?AWSAccessKeyId=<key>&Signature=kz30gf304b%2F%2F93pQeUSPrto5MiE%3D&x-amz-security-token=<token>&Expires=1601690152

  at ~/.pyenv/versions/3.8.5/lib/python3.8/site-packages/requests/models.py:941 in raise_for_status
      937│         elif 500 <= self.status_code < 600:
      938│             http_error_msg = u'%s Server Error: %s for url: %s' % (self.status_code, reason, self.url)
      939│ 
      940│         if http_error_msg:
    → 941│             raise HTTPError(http_error_msg, response=self)
      942│ 
      943│     def close(self):
      944│         
      945│         called the underlying ``raw`` object must not be accessed again.

If I change the following lines in the poetry code:

   2  ~/.pyenv/versions/3.8.5/lib/python3.8/site-packages/poetry/repositories/pypi_repository.py:454 in _download
       453│     def _download(self, url, dest):  # type: (str, str) -> None
     → 454│         return download_file(url, dest, session=self.session)

changes to:

   2  ~/.pyenv/versions/3.8.5/lib/python3.8/site-packages/poetry/repositories/pypi_repository.py:454 in _download
       453│     def _download(self, url, dest):  # type: (str, str) -> None
     → 454│         return download_file(url, dest)

and re-run, everything works:

$ poetry update
Skipping virtualenv creation, as specified in config file.
Updating dependencies
Resolving dependencies... (41.8s)

No dependencies to install or update

It seems like the auth is needed to talk to the API for package version resolution but causes issues when it is also used for package downloads. If it makes any difference, I am using pypicloud as the backend for my private PyPi installation. I am trying to be as brief as possible with my output as possible without dumping any keys or stuff like that. Please let me know if you need any more information or suggestions on what I should be changing in my configuration to get my stuff working again.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 3
Comments: 25 (9 by maintainers)

Commits related to this issue

improve http request handling for sources This change refactors HTTP repository source implementations. The following changes have been made. - CacheControl cache now lives within Authenticator. - A... — committed to abn/poetry by abn 2 years ago
improve http request handling for sources This change refactors HTTP repository source implementations. The following changes have been made. - CacheControl cache now lives within Authenticator. - A... — committed to abn/poetry by abn 2 years ago
improve http request handling for sources This change refactors HTTP repository source implementations. The following changes have been made. - CacheControl cache now lives within Authenticator. - A... — committed to abn/poetry by abn 2 years ago
improve http request handling for sources This change refactors HTTP repository source implementations. The following changes have been made. - CacheControl cache now lives within Authenticator. - A... — committed to abn/poetry by abn 2 years ago
improve http request handling for sources This change refactors HTTP repository source implementations. The following changes have been made. - CacheControl cache now lives within Authenticator. - A... — committed to abn/poetry by abn 2 years ago
improve http request handling for sources This change refactors HTTP repository source implementations. The following changes have been made. - CacheControl cache now lives within Authenticator. - A... — committed to python-poetry/poetry by abn 2 years ago

Most upvoted comments

@MasterNayru I would suggest that this issue should be considered as poetry issue. The reason being - same pypicloud server is working just fine with pip, and does not with poetry.

We run number of repositories and python packages, only one of them is with poetry and currently it does not work. We should either wait or help with fixing poetry or migrate to pip.

Katafalkas on Oct 23, 2020

@abn I am going to close this issue as it would appear that pypicloud have, since I last looked at it, defaulted to the behaviour which poetry is enforcing with regards to auth for package downloads. Rather than try to support the old behaviour which pip works with, I will update my pypicloud installation and make use of the new behaviour. Cheers for bearing with us.

I think we should consider reopening this issue as this problem still exists when using a pypi server that returns presigned urls to be used when fetching packages (for instance pypi-cloud)

Right now I’m forced to stick with poetry<1.1 due to this problem.

jensgustafsson on Jan 24, 2021

We hit exactly the same problem with poetry 1.1.1 and pypicloud 1.0.10. The way I understand it, this can’t be solved by pypicloud upgrade. Here’s my reasoning:

poetry makes a call to pypicloud
pypicloud returns a pre-signed URL pointing to our S3 bucket (the URL already contains AWS access key and token)
poetry makes a GET request (using requests library) but since it passes the session as @MasterNayru described in the issue, requests adds the Authorization header. This makes the request invalid. We verified it manually using curl.

Since it’s poetry/requests who adds the authorization header, I don’t see how this can be fixed on pypicloud side.

Please correct me (or reopen the issue 😉 ).

pepastach on Oct 6, 2020

Hi. Ran into the same issue. pypicloud==1.1.5 poetry==1.1.3

Tried both with and without pypi.stream_files = True - same issue.

Error is because of the headers being sent. The same url is downloadable using curl.

b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>Basic XxXx...</ArgumentValue><RequestId>58D6A864A3D27683</RequestId><HostId>FpXxx...</HostId></Error>'

Katafalkas on Oct 23, 2020

Great to hear that it is working for you. The setting you would have needed to set was storage.redirect_urls, not redirect_urls as you had it in the config you posted

Thanks! Yes I actually figured that out after your previous reply 🙏 Thanks a lot!

jensgustafsson on Jan 28, 2021

Also experiencing the same problem. We have tried to configure pypicloud according to suggestions in this issue, but no luck. It’s forcing us to stick to <1.1, which is unfortunate to leave all the great speed improvements of >1.1 on the table

FlippAre on Jan 27, 2021

@abn I am going to close this issue as it would appear that pypicloud have, since I last looked at it, defaulted to the behaviour which poetry is enforcing with regards to auth for package downloads. Rather than try to support the old behaviour which pip works with, I will update my pypicloud installation and make use of the new behaviour. Cheers for bearing with us.

Same for me, forced to stick with poetry<1.1 due to this problem.

rizerzero on Jan 25, 2021

This is how I setup private pypi

Edit pyproject.toml
[[tool.poetry.source]]
name = "myprivate_pypi"
url = "https://pypi.myprivate_pypi.com/simple/"
At terminal, add poetry config credentials for private_pypi poetry config http-basic.myprivate_pypi <username> <password>

Update lock with --no-update poetry lock --no-update

Add your library that is found at private pypi or poetry install poetry add <my-packate-found-at-private>

@meanderingcode let me know if this one works for you

@cereblanco Thank you. I discovered this yesterday when looking at changes related to legacy repositories. Edited my comment on your PR.

MeanderingCode on Dec 12, 2020

@abn I am going to close this issue as it would appear that pypicloud have, since I last looked at it, defaulted to the behaviour which poetry is enforcing with regards to auth for package downloads. Rather than try to support the old behaviour which pip works with, I will update my pypicloud installation and make use of the new behaviour. Cheers for bearing with us.

MasterNayru on Oct 2, 2020