poetry: poetry throws an error about SHA string length when trying to install a dep from a private repo using a token

Poetry version: Poetry (version 1.4.1)
Python version: Python: 3.9.15
OS version and name: macOS 13.2.1
pyproject.toml: https://gist.github.com/sg-s/07ae0c878981d15b5c22d2adc7d74f3d
I am on the latest stable Poetry version, installed using a recommended method.
I have searched the issues of this repo and believe that this is not a duplicate.
I have consulted the FAQ and blog for any relevant entries or release notes.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

what i’m trying to do

install a dependency that is a private repo on github, using a github token
i did the following:

poetry -vvv add git+https://{token}@github.com/my/private-repo/

what i expected to happen

it to install, or at least give me some error about access.

what i got instead

an error that the length of SHA string isn’t 20

base error


  AssertionError

  

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/pack.py:699 in _object_offset
       695│ 
       696│         Args:
       697│           sha: A *binary* SHA string. (20 characters long)_
       698│         """
    →  699│         assert len(sha) == 20
       700│         idx = ord(sha[:1])
       701│         if idx == 0:
       702│             start = 0
       703│         else:

detailed error

Using virtualenv: /Users/srinivas/Library/Caches/pypoetry/virtualenvs/ideas-python-utils-ibgcsbwi-py3.9
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS

  Stack trace:

  26  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  25  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 

  24  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 

  23  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code

  22  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  21  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/commands/base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

  20  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/commands/command.py:62 in execute
        60│ 
        61│         try:
     →  62│             return self.handle()
        63│         except KeyboardInterrupt:
        64│             return 1

  19  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/console/commands/add.py:160 in handle
       158│             return 0
       159│ 
     → 160│         requirements = self._determine_requirements(
       161│             packages,
       162│             allow_prereleases=self.option("allow-prereleases"),

  18  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/console/commands/init.py:384 in _determine_requirements
       382│ 
       383│         result = []
     → 384│         for requirement in self._parse_requirements(requires):
       385│             if "git" in requirement or "url" in requirement or "path" in requirement:
       386│                 result.append(requirement)

  17  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/console/commands/init.py:443 in _parse_requirements
       441│             cwd = Path.cwd()
       442│ 
     → 443│         return [
       444│             parse_dependency_specification(
       445│                 requirement=requirement,

  16  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/console/commands/init.py:444 in <listcomp>
       442│ 
       443│         return [
     → 444│             parse_dependency_specification(
       445│                 requirement=requirement,
       446│                 env=self.env if isinstance(self, EnvCommand) else None,

  15  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/dependency_specification.py:216 in parse_dependency_specification
       214│ 
       215│     specification = (
     → 216│         _parse_dependency_specification_url(requirement, env=env)
       217│         or _parse_dependency_specification_path(requirement, cwd=cwd)
       218│         or _parse_dependency_specification_simple(requirement)

  14  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/dependency_specification.py:68 in _parse_dependency_specification_url
        66│ 
        67│     if url_parsed.scheme in ["git+https", "git+ssh"]:
     →  68│         return _parse_dependency_specification_git_url(requirement, env)
        69│ 
        70│     if url_parsed.scheme in ["http", "https"]:

  13  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/dependency_specification.py:49 in _parse_dependency_specification_git_url
        47│ 
        48│     source_root = env.path.joinpath("src") if env else None
     →  49│     package = Provider.get_package_from_vcs(
        50│         "git",
        51│         url=url.url,

  12  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/provider.py:384 in get_package_from_vcs
        382│             raise ValueError(f"Unsupported VCS dependency {vcs}")
        383│ 
     →  384│         return _get_package_from_git(
        385│             url=url,
        386│             branch=branch,

  11  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/provider.py:112 in _get_package_from_git
        110│     source_root: Path | None = None,
        111│ ) -> Package:
     →  112│     source = Git.clone(
        113│         url=url,
        114│         source_root=source_root,

  10  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/vcs/git/backend.py:429 in clone
       427│         try:
       428│             if not cls.is_using_legacy_client():
     → 429│                 local = cls._clone(url=url, refspec=refspec, target=target)
       430│                 cls._clone_submodules(repo=local)
       431│                 return local

   9  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/vcs/git/backend.py:256 in _clone
       254│             local = Repo(str(target))
       255│ 
     → 256│         remote_refs = cls._fetch_remote_refs(url=url, local=local)
       257│ 
       258│         logger.debug(

   8  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/vcs/git/backend.py:201 in _fetch_remote_refs
       199│ 
       200│         with local:
     → 201│             result: FetchPackResult = client.fetch(
       202│                 path,
       203│                 local,

   7  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/client.py:785 in fetch
        783│                 path,
        784│                 determine_wants,
     →  785│                 target.get_graph_walker(),
        786│                 f.write,
        787│                 progress=progress,

   6  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/repo.py:554 in get_graph_walker
        552│         """
        553│         if heads is None:
     →  554│             heads = [
        555│                 sha
        556│                 for sha in self.refs.as_dict(b"refs/heads").values()

   5  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/repo.py:557 in <listcomp>
        555│                 sha
        556│                 for sha in self.refs.as_dict(b"refs/heads").values()
     →  557│                 if sha in self.object_store
        558│             ]
        559│         parents_provider = ParentsProvider(self.object_store)

   4  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/object_store.py:385 in __contains__
        383│         This method makes no distinction between loose and packed objects.
        384│         """
     →  385│         if self.contains_packed(sha) or self.contains_loose(sha):
        386│             return True
        387│         for alternate in self.alternates:

   3  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/object_store.py:374 in contains_packed
        372│         for pack in self.packs:
        373│             try:
     →  374│                 if sha in pack:
        375│                     return True
        376│             except PackFileDisappeared:

   2  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/pack.py:2370 in __contains__
       2368│         """Check whether this pack contains a particular SHA1."""
       2369│         try:
     → 2370│             self.index.object_offset(sha1)
       2371│             return True
       2372│         except KeyError:

   1  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/pack.py:686 in object_offset
        684│             sha = hex_to_sha(sha)
        685│         try:
     →  686│             return self._object_offset(sha)
        687│         except ValueError as exc:
        688│             closed = getattr(self._contents, "closed", None)

  AssertionError

  

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/pack.py:699 in _object_offset
       695│ 
       696│         Args:
       697│           sha: A *binary* SHA string. (20 characters long)_
       698│         """
    →  699│         assert len(sha) == 20
       700│         idx = ord(sha[:1])
       701│         if idx == 0:
       702│             start = 0
       703│         else:

About this issue

Original URL
State: open
Created a year ago
Comments: 21 (5 by maintainers)

Most upvoted comments

I’ve also run into this problem, I don’t have a reproduction, but I can leave a breadcrumb … it seems to be triggered when a repo that your other private repo depends on is already cloned into Poetry’s .venv/src folder. Deleting the specific repo that your private repo depends on from that folder and adding it again via the normal command seems to run.

timos-flwls on May 15, 2023

I’ve also run into this problem, I don’t have a reproduction, but I can leave a breadcrumb … it seems to be triggered when a repo that your other private repo depends on is already cloned into Poetry’s .venv/src folder. Deleting the specific repo that your private repo depends on from that folder and adding it again via the normal command seems to run.

I had a similar issue: poetry lock always failed with the exception mentioned in this issue whenever a given dependency from a public git repo (pytensor) was in pyproject.toml. So, in my case, it happened from a public repo (not PRIVATE!). The solution was the one quoted, thanks @timos-flwls : I just deleted .venv/src/pytensor and the problem disappeared.

Maybe, in my case, an initial poetry install was interrupted and left a corrupted/broken clone in .venv/src/pytensor and so all subsequent poetry lock failed. Clearly, this is hard to reproduce, unless you damage a previously cloned repo on purpose.

Maybe a dulwich issue more than a poetry issue?

All in all: remove the guilty repo in .venv/src/ if you stumble into this issue.

emanuele on Jul 13, 2023

I can confirm this bug still occurs in Poetry 1.7.1 as well. Maddening indeed.

techwizrd on Jan 21, 2024

Bug and resolution still exist with Poetry 1.6.1. Removing repo in ~/.cache/pypoetry/virtualenvs/<redacted>-py3.11/src allowed the poetry lock command to succeed. Error was consistent regardless of using the git repo’s full hash, the 11 char slug, or the first 20 chars. Maddening. Glad this thread exists.

gogobera on Nov 7, 2023

In my way, it reproduced like that:

In repo: make commit, push it (with some commit hash)
In poetry project: add repo {git = "path-to-repo", rev = "hash"}, run poetry lock
In repo make git reset HEAD~1, then make new commit on base of it, then force push the commit git push -f (it has some commit with new hash2)
In poetry project: change repo {git = "path-to-repo", rev = "hash2"}, run poetry lock, it has the error assert len(sha) == 20

I assume that older hash is deleted when we made a force push commit, so poetry couldn’t detect an older hash in github to compute SHA correctly. As it’s written above, removing .venv/ solves the problem

bobokvsky on Sep 18, 2023

I just experienced this error while doing a poetry lock with a public git repo and, similar to what @emanuele says, in my case it went away after I nuked the venv and recreated it.

  AssertionError

  at ~/.local/pipx/venvs/poetry/lib/python3.11/site-packages/dulwich/pack.py:698 in _object_offset
       694│ 
       695│         Args:
       696│           sha: A *binary* SHA string. (20 characters long)_
       697│         """
    →  698│         assert len(sha) == 20
       699│         idx = ord(sha[:1])
       700│         if idx == 0:
       701│             start = 0
       702│         else:

sacundim on Aug 2, 2023

What more info do you need to reproduce?

… a way to reproduce it!

If you can start from a fresh project and provide steps to show the error, please do so. Even better, wrap that up in a dockerfile to make sure that the same thing will happen for someone else.

poetry is just a regular python program, debug it however you like

dimbleby on Jun 18, 2023

this is a terrible way to debug and I am dropping out of the conversation.

I recommend that you close this as a duplicate of one of the others.

If you want to share your problem, then set up a repository that does not contain anything secret so that anyone can reproduce whatever it is you are seeing.

dimbleby on Mar 30, 2023

I did a poetry add with a private repo that I have on github and this did not reproduce. Without a way to reproduce, no-one but you can do anything with this.

But it’s still bound to be a duplicate of those others and can be closed as such.

dimbleby on Mar 30, 2023

presumably duplicate #6455, #7596, though without a way to repro no-one can do anything with this anyway

please close

dimbleby on Mar 30, 2023