go: cmd/go: modules with buggy releases
Consider the following case where you have an app with a dependency on a module (modA) that has a dependency on another module (modB).
App --> modA --> modB
modB is released with a bug. For the examples sake, the bug causes goroutines to not be garbage collected and the number to continue to grow. This bug has been experienced in a real world application.
The developer of modA realizes this bug exists and sets the minimum version to the last stable release that does not contain the bug.
The developer of the App decides to update the dependencies to the latest releases (e.g., go get -u ./...). Alternatively, the apps developer could import a second dependency (modC) that also depends on modB but at the latest release because they did not catch the bug. Some bugs don’t show up on tests but rather in long running situations which is why I choose the goroutine example.
There is currently no way for the author of modA to pass along information that modB has a bad releases. We need a way to communicate that information for those periods where that release is in the wild and one to use.
Existing tools like glide and dep let you say not to use a release. This is idea shows up in most other languages dependency management as well (e.g., !=1.2.3).
An idea was suggested, rather quickly and off the cuff, by @rsc to setup a web service to hold the information on buggy versions that should not be used. There are a few constraints that make this idea difficult:
- Who can post details on what versions of what tools are incompatible? There is an opportunity to troll through manipulation of the data
- How is access handled when packages can be hosted by VCS and you can’t always count on GitHub
- There is an information leakage problem where data on the dependencies of a proprietary app could be leaked to this services host. For example, a rival to Google is working on an app but leaking the dependencies they use to Google as a host of the service which gives lots of detail. We shouldn’t have a case where data is leaked like that
This is a problem because of two things:
- When not vendoring you lose the ability to easily diff the changes to dependencies
- A mass number of developers never really check the changes to their dependencies when they pull in updates
- A bug like this could be easily missed when updating, even for the trained eye, which I posit few will do
How will the Go toolchain handle this situation?
About this issue
- Original URL
- State: open
- Created 6 years ago
- Comments: 32 (19 by maintainers)
If this issue is just about marking releases as insecure / buggy, then this is a duplicate of https://github.com/golang/go/issues/24031#issuecomment-407798552 .
If this issue is about code review and diffing changes, then I have a few thoughts. I’d like make a to address some of the issues you mentioned with code review, but I don’t it should be part of
cmd/go. Maybe we could discuss something over in Athens tracker or offline first.If this issues is about the lack of a way for
go.modto transitively exclude module versions, then I think this is working as designed as you noted. One tool you could build could read all the mod files from dependencies and inform the user of any excludes constraints they request locally, and optionally include them locally. As you noted, this isn’t what you designed in glide and Russ has acknowledged this situation a number of times and it seems fundamental to the principal of the design.The workflow I’m envisioning for CI tools is, roughly:
The
go get -u=patchstep should fetch any updated exclusions and deprecations. Thego release -test=allstep (or however it ends up spelled) would run the tests and also report any detected incompatibilities.That doesn’t require any annotation mechanism beyond issuing a patch release with an exclusion in the
go.modfile, and doesn’t require any extra network access duringgo buildorgo test.proposal: inform the developer of any stated version exclusion used by dependencies at build time
This relies on the previous proposal XXX for fetching and reporting the list of modules. This pertains to the UX.
Whenever a
go buildorgo installis invoked, make a network request to determine if there has been any updates to all the dependencies in use exclusion list (implementation TBD). Alternatively, make thego buildorgo installcommand only periodically perform this check.One advantage of bundling it in with
go buildis that CI servers would contain this information in the log output, unless the CI server was isolated from the network or the implementation was periodic then it may or may not perform this check.Edit: to be clear, I’m trying to re-cap what I understand from the above conversation. This is not my proposal.
In other words, in that scenario outlined in my immediately prior comment:
By checking in and releasing a new modA with a new
go.mod, the author of modA is able to communicate to consumers of modA (in this case, the author of ‘App’) that already released versions of modA are incompatible with the buggy modB, without the author of modB needing to take any action, and without the author of ‘App’ needing to know they should try to get a new modA (because go tooling automatically checked the latest modA’sgo.modto discover the newly published incompatibility information wherein modA’s author is declaring modB to be buggy or declaring pair-wise incompatibility between certain versions of modA/modB).With this mechanism, this incompatibility information could be communicated using the same core mechanisms of releasing code (and hence the information moves with the same strengths and weaknesses of how the code itself is communicated).
In this example, it is the author of modA who both determines there is a problem and communicates the details of the problem, which is good because it is modA that directly depends on modB, and hence the author of modA is much better positioned to determine there is a bug in modB than the less-directly-connected author of ‘App’ (who is the integrator in this scenario, and who is integrating modB as an indirect dependency and as such the author of ‘App’ is likely not very aware of exactly how modA interacts with modB. Of all the different actors here, the author of ‘App’ is likely least well positioned to determine modB is buggy, though the author of ‘App’ might be the first actor here to experience the pain induced by a buggy modB when ‘App’ doesn’t work). The author of modA and the author of ‘App’ do not need to rely on any immediate action by the author of the buggy modB, which is good because maintainers can be busy or otherwise might not be able to act for days or weeks or months.
Ultimately, hopefully the story has a happy ending when the buggy modB is eventually fixed and released, but no one needed to wait for that to occur.
In any event, it seems the mechanism outlined in https://github.com/golang/go/issues/24031#issuecomment-407798552 could be used in the scenario outlined by @mattfarina here, but perhaps there is a much better or different solution that would be more appropriate.
edit: couple typos
@ianlancetaylor I’ve learned a little about this…
This is a good idea and would likely scale in data for some time. There are other problems in a system like this and I’m curious how those would be solved. Here are a couple that immediately come to mind: