poetry: Breaking change introduced in the legacy Poetry installer

@remram44 It’s still unclear what you mean/want. I’d still like input from @adzeeman, but in the meantime I’m going to suggest the following solution:

• By default, 1.2.0 will be selected and the installer will fail.
• Any explicit version can be selected, and the installer will succeed for versions it understands.
• If GET_POETRY_IGNORE_DEPRECATION=1, the latest installable version will be used even when no explicit version is selected.
• In all cases a deprecation message will be printed, and warnings will be issued when uninstallable versions of Poetry are overlooked.
• A target removal date of January 1, 2023 will be printed, with a issue to track the removal, and a blog post linking to that issue.

A brief reminder of how we got here: no arbitrary removal or cut-off for 1.2 was picked, instead (over a year ago) get-poetry.py was declared deprecated in the changelog, script itself, and on the blog. A warning was added for when it tried to install a version it did not understand. When 1.2 was released, the script started selecting 1.2 when it was instructed to install the latest version (i.e. no explicit --version). It was expected that users of the existing script (who had not used it in a CI environment with a pinned/explicit version) would have long migrated to install.python-poetry.org as the deprecation was advertised over a year ago.

The current changes are an attempt to drive awareness of the need to migrate independent from 1.2. I see the above as the best way to move forward from where we are now (given that there is wide-scale awareness of this now, it seems hopeful that January 1 won’t be another CI bloodbath).

I am very much opposed to installing the latest 1.1.x when no --version is provided without some sort of intervention, as we’re just going to repeat this come January 1 (or whatever target we set) – since anyone who cares has had to intervene in their pipelines (or is tolerating a random but low failure rate), GET_POETRY_IGNORE_DEPRECATION should already be present for users who do not want to pin a version anyway. The cat is already out of the bag there.

The version number based on SemVer doesn’t tell you anything about the amount of code changes, it is telling you something about the changes in the API. At Poetry we consider CLI as the API and nothing more.

Is it a problem if a hundred developers are using their own local poetry install while sharing repos? Anyone can lock and commit. Should everyone and all repos uses the same major/minor/patch or if I can expect some consistency? So far, it’s working well, but it’s all 1.1.x. I am asking because I originally thought that <2 was a reasonable pin, but now I think I need to do <1.2 instead for today, and eventually >1.1,<1.3… What do you recommend?

The installer script is a stand alone project.

To be totally honest with you, that’s not something users want to have to deal with.

I don’t feel like this kind of awareness is necessary in other CLI tools such as docker, kubectl, aws cli, etc. Either there are no rules, or they are very clearly typed out. Also, their old stuff pretty much always work forever when you follow their docs, it doesn’t suddenly break a 6-months old branch. They don’t assume that the reader knows about all of the best practices, so they take the time to type everything out if needed, and that’s why they are so successful.

The docs really should type out exactly what to do, such as:

POETRY_VERSION=1.2.0  # this is of course generated for the current version of the docs
curl -sSL https://github.com/python-poetry/install/get-poetry.py@some-static-sha | python -

So that there is no ambiguity. (edit: the link in my example is obviously wrong, it’s only to illustrate the concept).

This way you know that everyone actually pinned a static version, and can blame other people for not following the docs. That’s much better than having to do support only to spell out that not using @master was for us to realize and manage. Currently, you don’t know how people decided to pin their versions, and it causes great pain vs the backward compatibility of the installer, it would seem.

In light of the current events, I suggest that the documentation explains that the best thing to do in CI servers is to download the https://install.python-poetry.org script to a local repo and not to curl it like the documentation tells, because it’s not possible to pin the new installation script anymore (and you could steal my keys whenever you want!).

The reason why I did not switch to the new installer is to reduce the scope of changes. When I will be ready to use the new installer, I will take the time to make sure all the ends of our CI/CD pipeline can work with it, then I will be able to roll it out globally.

When I tried the new installer, I was immediately faced with a SSL issue because of an old public image. When I fixed that, I realized GitHub fails at poetry install with 1.2.0 but not with 1.1.15. Why? I don’t know; cache maybe? I don’t have time to test it at the moment. The old one works perfectly fine for my needs. There’s a “chain” there that must work with proofs and we’re talking multiple repositories and deployments. It’s trivial to do, but very time consuming. This process is usually reserved to clear vulnerabilities, so it’s pretty annoying when it’s just someone who codes an “if” to block some functionality for no critical reason.

You are right, I was foolish to curl @ master. I would have never done that if it wasn’t how the documentation was written in the first place (let’s be fair; the documentation recommends the --version, it never talks about not using @master). So I’ll just pin the commit before the brownout and call it a day. But now I’m scared about the new installer, and who knows what you’re going to do with it in 2 years. I mean, it’s already the 3rd installer… I think that proves that there’s a problem with the way you deal with backward compatibility.

What I would like to see here is the Poetry team acknowledging that this was poorly handled + mistakes were made, then offer a clean resolution that doesn’t require anyone to change or rewrite anything. In a corporate environment, this kind of situation would warrant an internal post-mortem and I really hope you will go through this exercise, as a team.

And while we’re at it; you should use SemVer too, and ensure that patches are low risk and security fixes, minors are new ADDED features and deprecation warnings, and majors come with deprecations.

I offer no view on whether the poetry project could have handled this better, but I want to encourage those of you who are blaming poetry for your CI problems to take at least a glance in the mirror.

Many of us have learned this the hard way so I have some sympathy, but the fact is: if you write your pipelines in such a way that strangers on the internet can break them - then sooner or later that will happen.

If you were relying on get-poetry.py never changing - you should have downloaded it and used your own copy. Count youselves lucky that no-one went rogue and uploaded a version that steals all your secrets and deletes all your code.

(But you shouldn’t use a local copy of the script either: because you’re getting all your dependencies from a repository that you own and not from public PyPI, right? So the script is useless to you anyway.)

Here endeth the lesson, please resume the complaining…

may I asked you, why you didn’t take the chance to replace the old installer by the new one? You can use the new installer to install 1.1.15 as well.

I can’t speak for others in this thread, but the looming problem on my team is that the install script is used in a docker container build pipeline. We maintain branches for old release versions so that we can build them if we need a patch. All of those branches contain a Docker build which will be invalidated by the removal of the installer. So this move doesn’t just force updating our current pipeline, it requires us to update the installer for every single release branch we maintain.

This is a bit useless after the fact, but I would like to point out that the way you advertised the deprecation of the old installer was not ideal. This is the warning that would be shown up the the day you made it fail on purpose:

This installer is deprecated, and cannot install Poetry 1.2.0a1 or newer.

What I read there is that “this installer is deprecated, what that means is it can’t install Poetry 1.2.0”. What you meant is that “this installer is deprecated and going away any minute, also incidentally it cannot install Poetry 1.2.0”.

Probably a lot of people, like me, were planning to upgrade to the new installer at the same time they upgraded to 1.2.0. All those people now have their pipelines broken because there was not a single day between the release of 1.2.0 and the day the “deprecated” installer was disabled (even if using it to install an older version with the --version flag).

Your “brownout” strategy is not tripping up stragglers who haven’t updated the installer despite many warnings. I am a very excited Poetry user who was going to update as fast as possible following 1.2.0’s release and was still trip up by this. Please consider revising your deprecation strategy (or having a public strategy…) for future breaking changes, it would be greatly appreciated.

I still believe that my original solution was the proper fix for the “planet”:

https://github.com/python-poetry/poetry/pull/6287

I have also spent some hours yesterday morning pinning some missing --version 1.1.15 to the legacy script in order to extinguish the fires. This morning, it started failing again even though I did pin the version (that 5% thing). So the “fix” that was introduced basically made our workaround useless in favor of a new workaround.

It would be better if the script would install the last compatible version (1.1.15) and issue a warning that to obtain 1.2+ you need to use the new installer, and even crash if you pass in --version 1.2.0 saying it’s just not the way to do it. That would be the best thing to do, because now, the worst that will happen is that people are going to bypass your deprecation with an env var, which has a good potential to create additional support cases about plugins not working. The worst that should happen, is that you successfully install and use an old version you’re used to 🤷🏻‍♂️

Train of thoughts:

• Deprecations and forced migrations are sometimes necessary, but it’s not in this case since the new installer is a new file with a shiny new url. There’s nothing to do to stay “backward compatible”, except preventing 1.2+ from being installed.
• The legacy installer could live at its documented URL forever and never be removed from the repository. You can probably even protect the file against changes. This is what we would expect from a “curl installable” script. There’s no rush to have everyone move to 1.2, since the 1.1.15 version works amazingly well and will most likely continue to do so for years. Feeling the urge to delete this file feels a bit OCD considering the harm it would cause.
• The 1.2 announcement mentioned “years of work” to come up to this. This sort of wording makes me think that this should have been released as 2.0. A major bump like that absolutely opens the door for people to have to migrate to the new installer.
• User experience is always the driving factor / priority. Some people have very complex build setups that takes hours, and also logging isn’t always easy to retrieve. It’s not very courteous to force a migration when the old version is still working just fine.

If you need 1.1, just pin to 1.1 – the new installer and supported install methods (pipx, manual) support 1.1 just fine, if you still need it.

did exactly that.

Since 1.2 stable is released yesterday. it is just a suggestion so that it won’t break for wide range of users.

Hi @adzeeman, sorry about the headache. To give you a little background, get-poetry.py was deprecated with a warning message for over a year (and the replacement was available for 6 months before the deprecation) – however usage of the script in production with no pinned version was much more common than we thought.

Thus, when 1.2.0 was released, the old installer would break as it would select the latest stable version for install, and then realize that 1.2.0 wasn’t installable. As a compromise it was suggested that the installer fall back to 1.1.x with loud warnings in that situation – we did so, but the with addition of a random failure percentage to try and force investigation and migration, since clearly warning messages, blog posts, and release notes had not made those downloading the script from our repo sufficiently aware.

I am open to the idea of considering an explicit --version to be an opt-out of the failure chance, but I hesitate because the eventual (total) removal of this script in several months will likely cause similar heartache. If we get it out of the way now, that may be for the best, even if it causes more pain and confusion now.

This was not meant to happen at the same time as 1.2.0 – honestly the plan was to get rid of get-poetry.py before the 1.2.0 release, but we simply didn’t have the bandwidth and the issue forced itself with the new stable release.

Since the cat is already out of the bag, do you think allowing for --version to silently continue is really any better since we still need to remove this installer? Note that the new installer has long been able to install Poetry 1.1 – the release branches and installers have no intrinsic ties, beyond 1.2 lacking the build machinery to produce get-poetry.py-compatible archives.

Finally, sorry again about the pain here – deprecating the get-poetry.py script was thought to have been a solved problem a year ago, though it is clear that the message the script printed for the last 12 months could have been more clear on the nature of the upcoming breakage. Future deprecations of similar tools (e.g. install-poetry.py in this repo) will be informed by this experience.