kubernetes: Building kubectl with Bazel as an external repo dependency is not currently possible.

What would you like to be added: I would like to be able to build kubectl as a dependency of another Bazel project. Specifically, it should be possible to add to a WORKSPACE file something like:

http_archive(
     name = "io_kubernetes",
     sha256 = "4737d3b10b27e391924f89b5ac3aaa5470619b92b75fe1ff5210cc30c56e2e53",
     strip_prefix = "kubernetes-1.10.10",
     urls = ["https://github.com/kubernetes/kubernetes/archive/v1.10.10.tar.gz"]
 )

load_something_something_kubernetes(...)

and then do bazel build @io_kubernetes//cmd/kubectl:kubectl

(note the load_something_something_kubernetes does not exist today)

Why is this needed: This would enable using kubectl as an external dependency of rules_k8s (https://github.com/bazelbuild/rules_k8s) project. This is very desirable as it would enable building kubectl from source (https://github.com/bazelbuild/rules_k8s/issues/227) and use that as the binary to use to execute k8s commands. We often get requests from rules_k8s users that this would be a very desirable feature.

More notes below expanding on what issues I found when I tried to build kubectl as an external dep:

I found the following issues:

the build depends on an @org_golang_x_tools repo. I could not find out where this is being pulled (perhaps in a repository_rule?), however, I could hack around this by copying it from the kubernetes project directly. I also needed to add manually to my WORKSPACE a http_archive rule to download @io_kubernetes_build
the repo is using genrules (e.g. vendor/k8s.io/apimachinery/pkg/util/sets/BUILD:23) which are running go binaries produced by other build steps, but are not referencing these properly when used in an external repo. I got this error message:

import "./vendor/k8s.io/apimachinery/pkg/util/sets/types": cannot find package "./vendor/k8s.io/apimachinery/pkg/util/sets/types" in:
        /usr/local/google/home/<...>/ffcac8bde6710ff680ff645bb4ae7c92/sandbox/linux-sandbox/30/execroot/io_bazel_rules_k8s/vendor/k8s.io/apimachinery/pkg/util/sets/types

fyi @fejta @chrislovecnm

/kind feature

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 1
Comments: 15 (12 by maintainers)

Most upvoted comments

Tested successfully with kubernetes-1.13.0-beta.1 (was able to build kubectl binary with kubernetes external repo) and all I needed to add to the rules_k8s workspace was:

http_archive(
    name = "io_kubernetes",
    sha256 = "dfb39ce36284c1ce228954ca12bf016c09be61e40a875e8af4fff84e116bd3a7",
    strip_prefix = "kubernetes-1.13.0-beta.1",
    urls = ["https://github.com/kubernetes/kubernetes/archive/v1.13.0-beta.1.tar.gz"],
)
http_archive(
    name = "io_kubernetes_build",
    sha256 = "21160531ea8a9a4001610223ad815622bf60671d308988c7057168a495a7e2e8",
    strip_prefix = "repo-infra-b4bc4f1552c7fc1d4654753ca9b0e5e13883429f",
    urls = ["https://github.com/kubernetes/repo-infra/archive/b4bc4f1552c7fc1d4654753ca9b0e5e13883429f.tar.gz"],
)

Note my workspace already included a recent version of rules_go.

Thanks for your help @ixdy . I’m closing this issue, but I still want to pursue doing the built + released kubectl binaries, so pointers to where those are located are still useful.

nlopezgi on Nov 21, 2018

That go_genrule issue in vendor/k8s.io/apimachinery/pkg/util/sets is hopefully fixed in 1.12+ (#64122) though I haven’t tested it in an external dependency setting.

ixdy on Nov 21, 2018

A few comments here:

The kubernetes 1.10 branch is pretty old, and depends on an even older version of rules_go. You might have an easier time with a more recent release, though a number of issues would still remain.
The various go dependencies come from rules_go with go_rules_dependencies() and go_register_toolchains(). We call those in our WORKSPACE.
In general, we discourage Go projects from vendoring kubernetes/kubernetes directly, and similarly we haven’t really put any effort into making kubernetes/kubernetes a valid bazel dependency, either. We’d probably need to factor out some of the dependencies from WORKSPACE like many projects do, and we’d also have to fix that genrule as you noted. I think @mikedanese opened a PR once to start fixing some of this, but it never merged.
Is there a reason you can’t depend on the built + released kubectl binaries instead, doing something similar to how rules_go downloads the correct toolchain?
I think kubectl intends to move to its own repo eventually, which should make this more manageable in the future.

ixdy on Nov 21, 2018