bazel: run_under=gdb not working with cc_test after Bazel 3.1.0 update

Can’t this be reverted to the old script until the new functionality that was supposed to go in is implemented properly? Adding new features to remove existing ones does not seem like a good idea to me

bazel build :foo && gdb "$(bazel info bazel-out)/foo" not going to work if the tests consume Bazel’s environment variables like TEST_TMPDIR.

I don’t think the bug is legitimately closed, version 3.1 introduced a regression that breaks people’s development cycle, it needs to be fixed, or at least marked as an open bug.

I don’t mind contributing back a fix, but I need your approval before implementing it, I saw you suggested a `–interactive which sounds legit, @bazel folks, wdyt?

Being able to debug a test interactively seems like a pretty crucial feature for non-build-system-developer usability. I would love an --interactive flag that makes it easy to do this. What’s the path something like that would need to take to get implemented? Is this something that would require a design doc?

Hey all - 9051faa was mine. Apologies for breaking functionality you were relying on.

The change breaks any test that is invoked via ‘bazel run’ and is expected to be interactive.

Is this a normal use-case? Bazel tests are not normally expected to be interactive. Quoth the documentation,

bazel run is similar, but not identical, to directly invoking the binary built by Bazel and its behavior is different depending on whether the binary to be invoked is a test or not. When the binary is not a test, the current working directory will be the runfiles tree of the binary. When the binary is a test, the current working directory will be the exec root and a good-faith attempt is made to replicate the environment tests are usually run in. <snip>

–run_under=command-prefix This has the same effect as the --run_under option for bazel test

And documentation from --run_under for bazel test:

Some caveats apply: stdin is not connected, so –run_under can’t be used for interactive commands.

Clearly the ‘stdin’ comment is now false, since #8322 from what I can find, but overall the documentation here seems consistent that this is not intended to be used for interactive commands, and emulates the environment of running the test, which seems to preclude doing something totally different for bazel run as I was about to suggest.

To be clear, I have no objection to the intended behaviour being changed, and interactive commands becoming a supported feature. I think that does run somewhat counter to the point of bazel run - why not use bazel build :foo && gdb bazel-out/foo in that case? - but as long as it’s possible to maintain bazel’s currently supported features while adding support for interactive wrappers (and that gets documented and regression tested), I see no reason not to add that to the mix.

In this case, it is clear that the documentation hasn’t evolved with the code (which often happens). That should not be the litmus test for whether a feature is support or not. As you mention, work was completed to support this feature in the commit you mention, as well as in others such as https://github.com/bazelbuild/bazel/issues/2815.

As for the usefulness of the command, it simplifies the process for users and doesn’t require them knowing the output location and name of the file to debug for build target. In fact, we use a config to wrap the run_under and point it to a tool that is acquired through bazel, making the process much simpler for our users.

Well, other than I don’t technically know how to implement it - non-bash wrapper seems required, but I don’t know if “interactive and attached to TTY” and “able to send its own children SIGINT” are even mutually possible across all supported OS.

Can’t this be reverted to the old script until the new functionality that was supposed to go in is implemented properly?

I’d prefer not - 9051faa fixes a fairly important core feature (getting test logs on timeouts), and as far as I can tell does not regress any supported or tested features. If we want to expand the set of features bazel supports, great, but that seems more appropriate as a fix-forward?

I disagree. You have functionality that has been part of bazel that has clearly regressed. Six members of the community have voted on the importance of this issue. Not only that, I don’t think that “does not regress any supported or tested feature” is a good argument. It does in fact cause a regression to changes that were explicitly committed to enable that functionality. In addition, Hyrum’s law is quoted in several bugs from committers stating a reluctance to break observable behaviour (e.g. https://github.com/bazelbuild/bazel/issues/5588#issuecomment-512913037)

Hello Everyone,

I’m thinking of contributing a solution based on a very lightweight “init” written in C. Something like “tini” might be a good starting point since it’s embedded into Docker and should be fairly straight forward to adapt to other UNIX too. https://github.com/krallin/tini

Would such a contribution be acceptable for this project? Are there any extra constraints that I should be aware of before beginning such an effort?

Thank you

Another workaround for you to consider. I validated this works for me for at least the simple use-cases I tried, though it’s not perfect (e.g. no scrollback history). Might suffice to unblock you for now? $ cat /dev/tty | bazel run --run_under=gdb :foo_test

It is our opinion that this is a bug: a regression in a feature that was explicitly added. The documentation was out of date and did not reflect the intended behaviour, and the outdated docs should not have been used as a justification to close this issue as “works as intended”.

I did some poking through history; looks like it’s pretty muddy. I can find no indication it was ever explicitly added as an intentional feature beyond simply happening to work as the test wrapper was originally implemented (“$@”), but it was known to work for this use-case, at least sometimes. Partially broken since Feb 2018 (bazel 0.12) by https://github.com/bazelbuild/bazel/commit/1001141f0674ff4b611814edcb00a5183680ef4a , explicitly not fixed for users hitting the “background subprocess” codepath per https://github.com/bazelbuild/bazel/issues/6145#issuecomment-425897684 , due to the presence of a workaround. stdin redirecting was incidentally fixed in https://github.com/bazelbuild/bazel/commit/8f22e94236e096508c531b4e48114c39489f0b1a# (bazel 1.1) for said codepath, but still for use-cases that didn’t need the kind of interactivity you describe.

@lberki , maybe you can comment on the supportedness or not of --run_under for interactive tools, since you were involved with #6145 ? Essentially, whether bazel run should be updated to prioritize interactive execution over perfect consistency with bazel test, or only when a new flag (--interactive) is provided, or not at all (workarounds only). Particularly for wrappers like gdb.

So the change https://github.com/bazelbuild/bazel/commit/9051faa313f29b8ad8ded68b48b3f7af13108d77 fundementally changes how the test process is run, effectively putting it into the background.

If said test is meant to be an interactive process, i.e. accessing the tty, then that process since it is in the background will be put into a STOPPED state by the kernel. The foreground process that has full control of the tty is the ‘test-setup.sh’ script itself. The change breaks any test that is invoked via ‘bazel run’ and is expected to be interactive.

I am seeing as well. In my case it is lldb, but the scenario is the same lldb is expecting user input and has gone in to a STOPPED state.

 PID   TT  STAT      TIME COMMAND
64689 s004  T      0:00.26 <Bazel-Path-here>/usr/bin/lldb

The test process needs to be in the foreground, for it to have full access to the tty and get user input, without being stopped.

There is a great explanation on process groups, session leaders and tty handoff in the response to this question here: https://unix.stackexchange.com/questions/507786/run-command-in-background-with-foreground-terminal-access

I am uncertain if this bash script as it is structured now can be changed to make interactive tests work again, there is allot going on in there. Moving the test process back into the foreground before disabling job control does allow interactive running to function again, but it changes things in that waiting for the child does not occur until the child exits, negating the wait on the child pid.

Maybe it is time to move to a native test-setup program that can handle both signal propagation, process groups, and proper tty handover?

Thanks @cmcgee1024 - this works with python breakpoint()

export PYTHONBREAKPOINT="pdb.set_trace"
cat /dev/tty | bazel run -c dbg //target:tests --run_under=/usr/bin/python3

This might suffice, but it has no readline support or true iteractive terminal behavior. For example, using ipdb or https://documen.tician.de/pudb/ has no readline support or interactions. If anyone has a workaround to get readline and other terminal support instead of the cat /dev/tty, would love to hear it.

@cmcgee1024 I’ll defer to someone on the bazel team to answer that; I can’t speak for what contributions they will/won’t accept. As long as it’s simple to maintain, and provides a reasonable story for other platforms (bazel supports linux/windows/mac, and I think runs on windows via some shell emulator), sounds reasonable; might be less work overall to pursue the --interactive flag as @wolfd offered though.

In either case, I haven’t seen traction from bazel folk on this bug for a long time, possibly because it’s closed. Consider opening a new bug for “add support for interactive tools to run_under”, and see if you get a better response there? It seems pretty reasonable to me to support this, via one path or another.

I can confirm that this workaround is functional when used with lldb on macOS with Bazel 4.1.0, but it doesn’t show a prompt ahead of time and it doesn’t use curses to make it look right in my terminal. Also, you need to hit enter an extra time at the end to make the OS shell come back with its own prompt. I expect that we will be putting this inside our bazel wrapper for all of our run commands and hopefully it doesn’t break at some point in the future.

$ cat /dev/tty | bazel run -c dbg --run_under=lldb :foo_test
...
quit
(lldb) quit

I think that it is important for Bazel to support interactive mode for a few reasons. It is managing the build and testing of a project hermetically, which makes it really difficult for any kind of manual intervention in the middle. The philosophy also lends itself to encapsulating all tooling into the Bazel ecosystem so that it can manage all of the dependencies. At some point, especially in development environments, there is a need to use interactive tools, such as a debugger, vm, emulator, etc. Also, as a developer, I want to be able to reproduce problems discovered in build or test as close to the same environment as I can while having the tools I need to fix the problem. This is why there really needs to be an interactive mode of some form. For example, docker has flags so that I can invoke an interactive shell inside a container that might be a stage in the build with either stdin or a pseudo-tty. It would make sense to have that kind of functionality here.

@EricBurnett any update on this one? I’ve used interactive bazel tests for debugging python tests using pdb which broken since 3.1.0 due to this commit: https://github.com/bazelbuild/bazel/commit/9051faa313f29b8ad8ded68b48b3f7af13108d77 @alanfalloon did you found a workaround for this issue?

Thanks, Moshe