poetry: Poetry cannot properly parse URL with Gitlab [deploy tokens]

Same here. This is a major blocker now.

@carlos-lm

Unfortunately, gitlab deploy token url does not work when one wants to use a command poetry add. We solved a problem by adding it straight to pyproject.toml and bypassing poetry add:

[tool.poetry.dependencies]
repo-name = {git = "https://gitlab+deploy-token-123:aabbddcceeff-ggh@git.example.com/repo_path/repo_name.git", tag = "1.0.0"}
# or
repo-name = {git = "https://gitlab+deploy-token-123:aabbddcceeff-ggh@git.example.com/repo_path/repo_name.git", rev = "aabbccdd"}
# or
repo-name = {git = "https://gitlab+deploy-token-123:aabbddcceeff-ggh@git.example.com/repo_path/repo_name.git", branch = "next"}

Error in our case is:

➜  poetry add git+https://gitlab+deploy-token-123:aabbddcceeff-ggh@git.example.com/repo_path/repo_name.git

  ValueError

  Invalid git url "git+https://gitlab+deploy-token-123:aabbddcceeff-ggh@git.example.com/repo_path/repo_name.git"

  at venv/lib/python3.8/site-packages/poetry/core/vcs/git.py:137 in parse
      133│                     groups.get("name"),
      134│                     groups.get("rev"),
      135│                 )
      136│
    → 137│         raise ValueError('Invalid git url "{}"'.format(url))
      138│
      139│     @property
      140│     def url(self):  # type: () -> str
      141│         return "{}{}{}{}{}".format(

➜ poetry -V
Poetry version 1.1.10

Installing using pip:

pip install git+https://gitlab+deploy-token-123:aabbddcceeff-ggh@git.example.com/repo_path/repo_name.git

works just fine

Any news here?!?

My Poetry versions are poetry=1.2.0a2, poetry-core=1.1.0a6.

In my case, I was able to use @EnriqueSoria’s workaround of using GitLab’s private PyPI repository, which allows me to use deploy tokens with poetry commands. Full details are as follows.

In the “source” project’s repository (the project which you want to install into one or more “destination” projects), put the following verbatim into .gitlab-ci.yml. (The CI job token exists automatically, and none of the variables need to be modified. Simply copy-paste! I assume that pyproject.toml is in the repo’s root.)

# A pipeline for uploading the project as a package to GitLab's private PyPI repository.
# (This allows the project to be installed from Poetry as a URL dependency with a deploy token
# which has `read_package_registry` scope.)

# References:
#  <https://docs.gitlab.com/ee/user/packages/pypi_repository/index.html#authenticate-with-a-ci-job-token>
#  <https://github.com/python-poetry/poetry/issues/2062>

build-wheel:
  when: manual
  image: python:latest
  script:
    - pip install twine
    - pip wheel --no-deps .
    - TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi *.whl

Now you can publish a wheel to “Packages & Registries” → “Package Registry”, by going to “CI/CD” → “Pipelines” and manually start the build-wheel pipeline.

To install this wheel to some “destination” project, in your destination project’s pyproject.toml you should put

[[tool.poetry.source]]
name = "identifier-for-gitlab-repository"
url = "https://gitlab+deploy-token-123456:t0k3n@gitlab.com/api/v4/projects/PROJECTID/packages/pypi/simple"

To adapt this snippet to your project, you will need the following pieces of information…

• A deploy token from the “source” project which has read_package_registry scope. This can be generated from the “source” project under “Settings” → “Repository” → “Deploy tokens”. The token consists of a username and (secret) password.

• The “source” project’s Project ID number. This can be found right under the project name on the project’s GitLab overview page. For me, it consists of 8 digits.

Then run the command

poetry add project-name --source=identifier-for-gitlab-repository

where project-name is the “source” project’s name, as it appears in pyproject.toml under [tool.poetry] as name = "...".

If all went well, this will install the package and create an entry in pyproject.toml of the form

[tool.poetry.dependencies]
project-name = {version = "^x.y.z", source = "identifier-for-gitlab-repository"}

Using poetry-core / pip install

I’m using Docker, and to keep the container slim, instead of poetry install which requires the full Poetry installation, I use pip install which uses the much slimmer poetry-core backend as per

[build-system]
requires = ["poetry-core>=1.0.8"]
build-backend = "poetry.core.masonry.api"

The downside is that pip is not aware of the tool.poetry.source section. But it a repo can be configured with the PIP_EXTRA_INDEX_URL envvar. To keep a single source of truth, I read the URL from the pyproject.toml file using a program called dasel:

COPY --from=ghcr.io/tomwright/dasel:v1.24.1-alpine /usr/local/bin/dasel /usr/local/bin/dasel
RUN PIP_EXTRA_INDEX_URL=$( \
        dasel select -f pyproject.toml -m \
        "tool.poetry.source.(name=identifier-for-gitlab-repository).url" \
    ) \
    pip install --editable .  # (The editable flag is not essential here.)

Removing the deploy token from pyproject.toml

It’s generally bad practice to keep secrets like deploy tokens in a file like pyproject.toml which is likely under version control. Poetry can manage the credentials for the repository. Simply run

poetry config repositories.identifier-for-gitlab-repository https://gitlab.com/api/v4/projects/PROJECTID/packages/pypi/simple
poetry config http-basic.identifier-for-gitlab-repository gitlab+deploy-token-123456 t0k3n

and then you can delete the gitlab+deploy-token-123456:t0k3n@ part of the url in pyproject.toml and everything will still work. (To test, delete the project-name = line from pyproject.toml and rerun the poetry add command.)

Troubleshooting

Make sure your deploy token has read_package_registry scope and not just read_package_registry.

@xinbinhuang would making use of gitcredentials be the better apporach here? This would work both in your development and deployment scenarios. As an additional benifit, would allow you to ensure seperation of privilleges using different credentials.

As for respecting the git url spec, I agree we should ensure that a url of the form https://user:pass@example.com/repo.git is accepted. However, I would not consider using these urls when adding dependencies the best practice. This is because that would mean that the credentials will end up in plain text in both your peotry.lock and pyproject.toml files. We should also considering logging warnings once this is fixed and a password is matched/detected.

A reasonable approach might be extend the URL validation to also include a new group “password”. The user should stil consider : invalid.

Same issue here. poetry install fails in Gitlab CI with a package added with Gitlab Deploy Token. Locally it works fine. I use Python 3.9 and Poetry version 1.1.11 in both environments.

For me it works (installing from private pypi) if I expose my token in pyproject.toml

[[tool.poetry.source]]
name = "<a_name>"
url = "https://<token_name>:<token>@gitlab.com/api/v4/projects/<project_id>/packages/pypi/simple"

@abn Thanks for the suggestion, I have not used gitcredentials before.

While I think your suggestion is valid in common scenarios for user passwords, it is a different case for Gitlab Deploy Tokens. These deploy tokens are not user credentials and designed specifically by Gitlab with limited permission scopes (similar to service account). So in the context of Gitalb Deploy Tokens, I believe it’s valid.

While security suggestion by warnings is good (with optional disabling it), I don’t think poetry as a package management tool should by any means enforce this validation and should leave this as a choice for the users.

I haven’t used Gitlab for a while now, so it doesn’t affect me that much. I like poetry and hope it will be successful in the future, so I hope poetry can make good decisions along the way.

I still run into the same error on gitlab CI:

Locally it works. Both used the same versions: Python 3.8, poetry v1.1.10

@EnriqueSoria This is for PyPi packages and not git source installation, isn’t it?

@jedie , while waiting for the bug to be fixed. You can pin your poetry version in your pyproject.toml

For example

[tool.poetry]
name = "new-package"
version = "0.1.0"
description = ""

...

[build-system]
requires = ["poetry<=0.12"]                 # this line: pin poetry to be <= 1.0.2
build-backend = "poetry.masonry.api"