bazel: Gracefully filter targets not compatible with the current Platform
There was a discussion recently on bazel-discuss about selectively building targets based on a platform: https://groups.google.com/forum/#!topic/bazel-discuss/U6sFbPWiXGM
Greg Estren suggested:
Ultimately we should integrate this with Bazel’s upcoming platform semantics (which builds on the concepts above) to provide as much granularity and flexibility as you want.
This is a request to do exactly that. I’d like to be able to say “build for this platform, but gracefully exclude targets that aren’t compatible” (where gracefully means not failing the build). Perhaps a --filter_unsupported_platforms flag? This would let a CI script do something conceptually like for platform in [list of supported platforms]; bazel build --platform=$platform --filter_unsupported_platforms //...; done, while maintaining targets in the repository that aren’t compatible with all platforms.
A suggested alternative in that thread is to use --build_tag_filters/--test_tag_filters. I’ve found this to be clunky because those flags don’t accumulate when passed multiple times - if you have configurations in tools/bazel.rc that set these flags, you can’t subsequently add another copy of --{build/test}_tag_filters on the command-line, or the value from bazel.rc will be overwritten.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 12
- Comments: 38 (28 by maintainers)
Just a quick update: @philsc is working on implementation now!
I’ve had a few detailed email conversations with @katre about this, to the point where I’m interested in speccing up a brainstorm proposal of what it might look like.
With Bazel’s platform integration steadily humming along, now may be a good time to follow through on this. If there’s still interest. There are still some unresolved design considerations, hence the desire for the brainstorm proposal.
If anyone currently wants this, please ping here.
If anyone wants to be part of the design discussion, also ping here.
@philsc 's PR just merged: https://github.com/bazelbuild/bazel/pull/10945. Congrats @philsc!
As for deployment, it looks like this’ll just make the cut to get into the next release?
We’re especially interested in your experiences and feedback using this once available. Also note @philsc and @AustinSchuh are delivering a virtual tech talk on this at BazelCon on November 13. Check that out for a good overview.
@jtattermusch , we are working on a real solution. It is proving to be hard, so progress is slower than we’d like. Greg is proposing above a workaround until this lands. In all 3 of the decently sized Bazel codebases I’ve used, all 3 have needed target skipping. https://docs.google.com/document/d/1cjR8oUwGAtokKbpBJR-jp26l2HxfOrKRJe20VZ7Rbvw/edit?ts=5d9c8924 has our attempt to document the problem statement. We would appreciate any feedback on it. gRPC is one of the projects that I used as an example in that document. Using gRPC has also helped me refine my views on how expressive the interface needs to be to work in the real world.
There is a follow-on proposal that is being worked on to propose an implementation which will allow for platform and toolchain based skipping and selecting. I can’t promise a timeframe, but it is being worked on. We’ll share it on bazel-dev when it is ready for feedback.
Here is a specific problem we are trying to solve with the `target_compatible_with" attribute:
We have thousands of “output” targets (binaries, test_binaries, packages) most of which are compatible with the tree standard platforms: Linux, Darwin, Windows_64. I would rather not visit them all and add them standard “target_compatible_with” attribute – too much hassle and code bloat. Few of those targets are special and compatible to Windows only – they should be skipped on any other platform and we want to mark them individually. Very few of these targets are special in the other direction – they are compatible with Emscripten target platform, additionally to the tree standard platforms. Again we don’t mind marking them individually. What I would like to avoid though is the explicit marking of the bulk of the targets in exactly the same way. Maybe we can introduce a configurable default value for “target_compatible_with” for the package, workspace, or simply in .bazelrc which we should be able to change on the target level. I would hate to visit each and every target to add that it is not compatible with Emscripten.
We need filegroup to accept a mix of the targets compatible and incompatible with the currently requested pattern and let ignore incompatible targets and collect compatible ones. But there is more to it – we also have “test_suite” rules, which accept the collection of the test target. Extra complexity here stems from the fact that for “test_suite” srcs attribute is not configurable!
Based on all that custom filtering rule does not look like a feasible path forward. It looks like we need Bazel built-in function to do the filtering and dependency on incompatible target through that function should not be considered invalidating dependency itself.