moby: non-standard vendor/ approach breaks building with Buildroot

Description

https://github.com/moby/moby/pull/44615#issuecomment-1343519928 https://github.com/moby/moby/issues/44614

The old releases pre-v23.x worked correctly with GO111MODULE=on.

The new releases which use a custom approach to managing vendor/ do not:

$ go build -mod=vendor github.com/docker/docker/cmd/dockerd

go: inconsistent vendoring in docker-engine-:
  cloud.google.com/go@v0.102.1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
  cloud.google.com/go/compute@v1.7.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
  cloud.google.com/go/logging@v1.4.2: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
  code.cloudfoundry.org/clock@v1.0.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
  [snip]

  To ignore the vendor directory, use -mod=readonly or -mod=mod.
  To sync the vendor directory, run:
  go mod vendor

Disabling go modules with GO111MODULE=off doesn’t fix it either:

After applying the following changes to buildroot:

  • Set GO111MODULE=off
  • Remove -modcacherw flag
  • Change path passed to go build from the module path to ./cmd/dockerd.

It still doesn’t work, since Go doesn’t recognize vendor/ anymore:

$ cd ./path/to/docker-engine
$ GO111MODULE=off go build -v ./cmd/dockerd
cmd/dockerd/daemon_unix.go:22:2: cannot find package "golang.org/x/sys/unix" in any of:
        /lib/go/src/golang.org/x/sys/unix (from $GOROOT)
        /host/share/go-path/src/golang.org/x/sys/unix (from $GOPATH)

The buildroot build infrastructure is here:

My request is to please play nice with the Go modules system and either go back to the way this worked before (which worked perfectly) or provide some solution for us which doesn’t involve completely changing the Go package infrastructure in Buildroot for a single package.

Reproduce

  1. Try to build with GO111MODULE=off

Expected behavior

No response

docker version

N/A

docker info

N/A

Additional Info

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (9 by maintainers)

Commits related to this issue

Most upvoted comments

go modules solved “some” issues, but in doing so effectively broke backward compatibility. We still have a LOT of work to do to make this happen (including a rename); we also need to refactor and restructure many of our packages to reduce dependency hell for consumers of the repository (before go modules, it was easier to ignore dependencies that you didn’t need, with go modules, all of our dependencies (versions) are enforces on users, which can complicate things as we also have some circular dependencies). But we want to make that transition, as without being a go modules, users of our code as a dependency must jump through a lot of hoops to be able to use it.

But yeah, go modules cause many breaking changes (they’re not considered “breaking change” as it’s outside of Go’s v1 compatiblity promise);

  • before go modules, go get <some project> always meant “latest from the default branch”
  • while some users used vendoring tools to “pin” to specific versions, Go itself had no versioning for dependencies (implicit convention was that “main” / “master” should always be backward compatible)
  • some projects started tagging releases to provide users a more convenient / semantic way to pin to a specific version, but that was not a requirement.
  • other projects (such as docker (now “moby”)) tagged binary releases; tags on this repository are for binary releases of docker / moby.

When go modules were introduced, this all changed:

  • Go took ownership of tags of all Go projects on GitHub (and elsewhere)
  • It enforces the uses of SemVer versioning, and assumes that any tag that “looks” like SemVer is SemVer
  • Tags that do not look like SemVer are ignored
  • Actually valid SemVer tags (SemVer does not officially allow v in versions) are considered invalid.
  • Release branches / long term release branches are not supported (versions may be rolled back to older versions that are found on the main/master branch).
  • go get <some project> no longer defaults to “latest from main / master”, but “latest tag that looks like SemVer”; which in some case means: dependencies get rolled back to a tag from years ago.
  • Any tag that is v2.0.0 or up is considered “incompatible”, and requires the project / import paths to be renamed. (in our case, requires all import paths to be renamed to “github.com/docker/docker/v20”)

go modules allow for “invalid” versions, as long as there’s no go.mod file in the repository; the moment that file is added, it is no longer importable (which is why we’re using vendor.mod).

+3
thaJeztah on Dec 10, 2022
Read more comments on GitHub
← moby: No route to host / connection refused - swarm mode on ARM
moby: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/registry\": executable file not found in $PATH": unknown. →