ruff: Ruff should consider adding rules first as a warning instead of a block

First of all, thank you so much for working so hard to create and maintain Ruff, this is an amazing project and it has made our CI so much faster than using flak8 and other tools. This issue is not a bug, but a behavior that Ruff should consider before adding new rules/checks.

In this latest version, Ruff added the PLR1714 rule and broke our CI because the project I work on has code that this rule complains about - https://github.com/aws-powertools/powertools-lambda-python/pull/2822. In my opinion, this can be considered a breaking change as it has changed the behavior of how Ruff checks our code and evaluates rules.

That said, I would suggest that Ruff initially add a new rule/check as a warning for at least 2 or 3 releases before considering it a block, so we have time to fix the problems before breaking the CI.

Please let me know if I’m missing anything, but you should take this into account to improve the developer experience for Ruff customers.

Thanks

About this issue

Original URL
State: closed
Created a year ago
Reactions: 2
Comments: 16 (9 by maintainers)

Most upvoted comments

We’ve had a few discussions around this internally and have some initial ideas on what a more useful versioning scheme could look like. Our tentative plan is to queue this up as a project for next month – so we’ll draft a proposal internally, circulate it for public feedback and discussion, then put a transition plan in place.

charliermarsh on Aug 11, 2023

I agree that ruff’s versioning scheme does not convey a lot of information. Even with ZeroVer one could distinguish between feature (potentially breaking) and bugfix releases. Is there a reason for only bumping patch versions?

However, I think it’s important to be precise and in my opinion adding a rule is not a breaking change. A breaking change would mean dropping or changing the meaning of CLI or configuration options. A new rule might prevent a pipeline from passing, but that’s due to a changed return code of the ruff invocation and not due to a break in ruff’s API. That’s also how Pylint is versioned.

Mr-Pepe on Jul 25, 2023

I think this is an inherent “problem” of linters. What if a rule has bugs and does not identify problematic code? The bug gets fixed and suddenly your pipeline breaks because previously unidentified code smells are found. In my opinion, it’s on the user to at least explicitly control the applied rules or (even better) to pin a version and explicitly update it. This also means that you have control over when your pipeline breaks and not your linter’s release schedule.

Mr-Pepe on Jul 25, 2023

@leandrodamascena - Heads up: we now have a proposal up for our versioning scheme available in https://github.com/astral-sh/ruff/discussions/6998

charliermarsh on Sep 1, 2023

Thank you for bringing it up, it’s an important thing that we keep meaning to spend time on 😅

charliermarsh on Aug 8, 2023

Yes, I’ll add it to my queue to post some thoughts here before the end of the week.

charliermarsh on Aug 8, 2023

In my opinion, it’s on the user to at least explicitly control the applied rules or (even better) to pin a version and explicitly update it.

For what it is worth, Rome recommends doing this on their Getting Started and Versioning pages.

tjkuson on Jul 25, 2023

Thanks for raising this problem, and I’m sorry that we broke your CI.

Here are some thoughts of mine on this:

We never defined our versioning policy explicitly listing what ruff considers a breaking change. An explicit versioning policy would set expectations and help users understand the risk of laxer version ranges.
Releasing new rules with warning severity helps users by giving them a grace period for upgrading. However, it doesn’t solve the breaking change problem by itself except if we decide only to introduce new rules when bumping the major version.

MichaReiser on Jul 24, 2023