cargo: Improve contributor experience by labeling complexity, ambiguity, and consensus of an issue

This is based on feedback from @jonhoo during team brainstorming on Zulip. Made it an issue for visibility.

The problem

The existing labels used to call for participations do not adequately describe the complexity of involved in the progress of staring from an issue to a pull request and merging it. Contributors and maintainers often suffer from this. For instance,

https://github.com/rust-lang/cargo/pull/11786 and https://github.com/rust-lang/cargo/pull/11775 are cases I failed to tell them the consensus and complexity beforehand.

https://github.com/rust-lang/cargo/issues/9599 is a case that the level of design and implementation complexity were embedded in comments.

https://github.com/rust-lang/cargo/issues/11666 is an example that implementation might be relatively easy but need to reach a consensus first.

https://github.com/rust-lang/cargo/issues/11432 is an example that the contributors saw the triage and thought “hmm… does that mean feature get accepted?”

This old https://github.com/rust-lang/cargo/issues/2526 may need an RFC, but is probably more dying for a driver taking care of it for a long while.

Every contributor has different expectations when helping the project. Not everyone can be satisfied by completing E-easy tasks. Also, not everyone loves discussing a design back-and-forth with the risk of proposal rejection. It could save both maintainers and contributors’ time if we have had a better labeling system around contributor participation.

What we have today

I said the contributor experience of each issue cannot be precisely described by existing labels. However you’re not sold. Let’s see what we have today.

We have three levels of participation experience. It’s not easy to tell if it indicates the discussion is easy or the coding part is hard.

Two labels calling out for design works. We don’t know how difficult to disambiguate the requirements into a proper design.

In terms of consensus, we have:

Apart from I-nominated. All other labels seems more like “there is a consensus” instead of “what is the level of consensus here?” or “how risky is my work to get rejected?”.

Factors affecting contributor experience

Jon’s original text summarizes these aspects down to three factors: complexity, ambiguity, and consensus. I copied the original words from him verbatim. Click to expand for details.

Complexity: how hard is this issue to fix/feature to implement in terms of just code adjustments.

This focuses solely on the “mechanistic” aspects of the change: how many places need to be changed, how involved are those changes, how many knock-on changes do we expect, etc. A complexity-low issue would only require changing one or two files (ideally just one or two functions) in fairly straightforward ways, or if there are more widespread knock-on changes they are entirely deterministic and could (ideally) be performed by search-replace. A complexity-high issue on the other hand is likely to require involved changes across multiple points in the code-base, and those changes are in turn likely to require non-trivial changes to other parts of the code that interface with the first set.

Ambiguity: how much design work is necessary to land the necessary change.

This is all about the degree to which the bug/feature is un-/under-specified and requires active decision-making — to what extent will a prospective contributor have to sit down and think/consider how to best fix this issue/implement this feature. An ambiguity-low issue should require very few decisions, if any, to be made. A contributor should be able to go make “the changes”, and it should be relatively well-understood (or obvious) what “the changes” are. An ambiguity-high issue on the other hand will require a signifiant exploration of the problem domain and several decisions will have to be made as part of the implementation. Those decisions need to be documented, and a rationale given, in case a subset of them need to be discussed or changed. A a contributor to such an issue is also going to have to think about implications beyond just the change they’re making, such as how it interacts with other project features, backwards-compatibility, etc. Very high ambiguity issues are likely to require an RFC.

Consensus: how comfortable are the core project maintainers with the proposed path forward.

This is essentially scoring “how likely is it that this will land without too much discussion after I’ve done the work”. Alternatively, it can be thought of as how controversial the change is. Unlike the other two, -high here implies “easier” and -low implies “harder”. It is also generally going to be inversely proportional to the ambiguity score, but not always. A consensus-high issue is one where little discussion is likely to have to happen once a PR is made. It’ll still have to go through review and such, but it’s unlikely that the fundamental approach is going to be questioned or endlessly debated. A consensus-low issue on the other hand is likely to require extensive debate before landing, and has a real danger of not landing even with a perfect implementation because of considerations beyond the quality of the feature or bugfix itself. PRs for consensus-low issues will almost certainly require an FCP.

Labels indicating these factors

The proposal above proposes 9 labels for these factors:

  • Complexity-high
  • Complexity-medium
  • Complexity-low
  • Ambiguity-high
  • Ambiguity-medium
  • Ambiguity-low
  • Consensus-high
  • Consensus-medium
  • Consensus-low

It is not clear how these labels fit in the existing label system in the wider Rust community. We could probably make E-easy means Complexity-easy. We could also make Ambiguity-* become E-ambiguity-*, so that contributors can figure out “Oh! I expected to experience more discussions if working on that.” Consensus-* may indicate that maintainers would have to get a conclusion, so they may fit in I-consensus-* category.

But perhaps they should all fit in E-* so people know they are all about contributor experience.

When will it start

I do think this kind of granular labels are invaluable. It is also a two-way door decision — it can always easily be reverted. The only stuff I am not comfortable is naming and how to fit in the current system (and also choosing label colors). So, if any of the team member give me a green light. I may just try one approach and see how far it goes.

Additional information

We could also add a E-needs-driver to indicate that we need a mid- to long-term driver for a feature/issue, though this is slightly tangent to the issue.

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 3
  • Comments: 21 (19 by maintainers)

Commits related to this issue

Most upvoted comments

I’d like to propose the following which is a bit of a mixture of what is suggested above and my own experience with dealing with issues.

My primary goal is to have a clear status for an issue, and how it can move further along.

Current landscape of labels

Just to recap what is used today:

Status

Experience

Importance

Priority

Categories

Misc

RFC Bot

Proposed changes

Add a clear set of statuses

Every issue should have a clear answer as to “what is the status of this” or “what is it waiting on”. Usually there should only be one status label (though that is not a requirement).

The following are not in a strictly linear set of steps, but are roughly the direction things flow. People should apply whichever S- label is appropriate.

Propose to add the following:

  • S-triage — This issue is waiting on initial triage.

    This should be auto-added to all new issues.

  • S-needs-info — Needs more info, such as a reproduction or more background for a feature request.

    Anyone is welcome to help with providing additional info to help reproduce or provide more detail on use cases and such. But usually this is a request to the initial author.

    When adding this label, there should also usually be a comment that goes along with it stating the information requested.

  • S-needs-team-input — Needs input from team on whether/how to proceed.

    Here it is essentially blocked waiting for a team member to move it to the next stage.

  • S-needs-design — Needs someone to work further on the design for the feature or fix.

    Anyone is welcome to help at this stage, but it should be clear that it is not yet accepted. It is expected that people should contribute comments and ideas to the issue which furthers the process of fleshing out what is needed, or alternate ideas. This may also require reaching out to the wider community via forums and such.

  • S-needs-rfc — Needs an RFC before this can make more progress.

    Anyone is welcome to help at this stage, but it should be clear that it is not yet accepted. However, this should only be tagged for changes that are somewhat likely to be accepted.

  • S-needs-mentor — Needs a team member to commit to helping and reviewing.

    This is for something that is accepted, such as after an RFC or a team discussion, or an obvious issue that just needs fixing, but no team member is available to help or review.

  • S-accepted — Issue or feature is accepted, and has a team member available to help mentor or review.

  • S-waiting-on-feedback — An implemented feature is waiting on community feedback for bugs or design concerns.

    Most tracking issues should go into this state after they have been implemented.

The E-* experience labels should be added at any stage to indicate the expected difficulty if it is known.

If someone wants pointers on where they can help, it depends on what they are interested in:

  • S-accepted — Anyone should be able to jump in depending on their skill level and interest (and it is not already assigned).
  • S-needs-design or S-needs-rfc — If you are interested in doing design work, these need someone to help move it along.
  • S-triage — Anyone can help with answering questions, asking for more detail, helping to reproduce, adding labels (like A-*), etc.

For tracking issues, they should either be S-needs-mentor, S-accepted, or S-waiting-on-feedback. For a partially implemented feature, it can have both S-waiting-on-feedback (for what is implemented) and S-needs-mentor or S-accepted to finish the rest of the work.

Other label changes

+3
ehuss on Apr 17, 2023

Something that came up in the cargo meeting is distinguishing between

  • “accepted for impl” its a problem but not one for random contributors to jump in on. Only for someone really ambitious / desperate. These would be under S-
  • “mentor-ready” where a team member is ready to discuss / review. These would be under E-
+2
epage on Mar 7, 2023

@weihanglo Are there any other things you’d like to see done for this, or can this issue be closed out?

+1
ehuss on Apr 20, 2023

I am not sure what color for S-needs-mentor should we use. If we want to make it obvious that it’s accepted, I’ll go with the same color of S-accepted.

I don’t have a strong opinion either way. Another option is to make it orange or yellow to be somewhere in-between green and red to correlate with it being somewhere between one of the red ones and being accepted.

Should we rename S-needs-mentor to S-accepted-needs-mentor so that it is clearer on its acceptance.

No strong opinion here, either. I slightly lean towards the shorter form, but either is fine with me.

+1
ehuss on Apr 19, 2023

Since the team is happy with Eric’s proposal, I am going to list the following tasks one by one. Feel free to take over any if it hasn’t yet assigned to anyone.

  • Review existing issues that state needing or accepted are still so. @weihanglo
  • Migrate all old labels and create new ones. @weihanglo
  • Make S-triage autolabels for all new issues. @ehuss
  • Document the new issue labeling mechanism in Cargo Contributor Guide. @ehuss
+1
weihanglo on Apr 20, 2023

Do all S-* share with the same color?

No, I think we can use different colors as needed.

I think the current E-help-wanted and Feature accepted labels should be turned into S-needs-mentor, not S-accepted, unless they are specifically simple or already have a Cargo team member available to help/review.

Yea, we definitely should review the existing issues before we migrate them. In practice we have been using those as wanting someone to start on them, but we should make sure that is still the case.

+1
ehuss on Apr 18, 2023

One suggestion: I think the current E-help-wanted and Feature accepted labels should be turned into S-needs-mentor, not S-accepted, unless they are specifically simple or already have a Cargo team member available to help/review.

+1
jonhoo on Apr 18, 2023

The original idea of this proposal is “maintainers add labels and contributors evaluate the risk and time if working on it”. The criteria of adding those labels need not be strictly consistent among maintainers. They are just “hints” for potential contributors.

I was speaking from a contributor angle. They would need a single label as an entry point (and discover it), plus 3 effort labels, plus at least 1 category and 1 area label. This means there would be 6+ labels on any given issue which turns it into a soup. You could drop the entry point label but now a user has to discover and decipher 1 of 9 labels as their entry point for possible issues and then walk through the other ones until. they find an issue of interest.

+1
epage on Mar 3, 2023
Read more comments on GitHub
← cargo: Improve `cargo add -F crate/invalid_feat` error discovery
cargo: Improve error message when there's a colon in the project directory name →