googletest: add timeouts to tests

We are not going to add timeouts to tests. This can be accomplished in many ways and there is very little benefit to support it in the framework itself

+1 I think it’s an important feature. I currently run tests under Linux’s timeout command, which is a crude way around this issue.

Answering @BillyDonahue - here’s how I would implement this feature:

• Forking seems inevitable. So fork the main program before running any tests.
• The child runs tests, the parent monitor its progress.
• A shared memory area would be used to track the child’s progress - what test is currently running, when it started and the expected end time. Use a simple structure that works lock-free.
• The parent would occasionally check if the current test has exceeded its expected end time. If yes - kill the child and exit.

I think this also has implications for any project running CI. With timeouts in GT, the continuous integration output would still reach the end and they would know precisely which function deadlocked. Otherwise, they just see that their CI system ran for too long.

Hi people! I am looking for something similar mentioned by @daravi --gtest_timeout=5000 as I am building an automated test infrastructure which runs the tests on VMs with various OS (Linux/Windows). The tests require our USB device connected to the PC - which I achieved by sharing via USB redirector. All works fine which is fantastic but…

Unfortunately, I trusted developers that they write robust automated tests (which do not hang - lock up/waiting for user input) but unfortunately everyone is a human and makes mistaked 😅 Though it makes my interactive session hang forever and I cannot get tests output back even if I kill the session.

What I tried to implement was to run the tests individually and give them timeout (scheduling them on a Job) but unfortunately it does not close the device gracefully (sometimes the device is left in the unresponsive state if we just kill the process - needs reconnecting to reset the state which I cannot afford time-wise (we have thousands of tests to run)). Hence now I am hoping to implement it within the tests framework itself as you do not have --gtest_timeout=xxx option - currently I am looking at @pinventado solution (thanks for sharing!) - I will give it a go now.

However I just wanted to add up to the request - it would be cool to get this option into your framework 😃 We use it not only for (quick) unit tests but also for integration and system tests with real devices (as described above) and it is extremely useful to us. Your framework is very powerful and I am glad we could use it but it could be even cool-er 😏 I believe, at least for my case, I would need a cleanup task (like current Teardown()) which would clean up the device state before terminating.

Cheers!

Is it not possible to add --gtest_timeout=5000 option? (for setting maximum test case run-time for all test cases to 5000 milliseconds)

This will help avoid a test-case running forever and spamming the console. (which makes it hard to find out which test case is failing because all google test output messages are cleared)

Hmm this is odd, I copied the link to my previous post but it gave a link to another. Anyway, this is what I actually meant … Create a new macro to address timeouts.

#define ASSERT_DURATION_LE(secs, stmt) { \
  std::promise<bool> completed; \
  auto stmt_future = completed.get_future(); \
  std::thread([&](std::promise<bool>& completed) { \
    stmt; \
    completed.set_value(true); \
  }, std::ref(completed)).detach(); \
  if(stmt_future.wait_for(std::chrono::seconds(secs)) == std::future_status::timeout) \
    GTEST_FATAL_FAILURE_("       timed out (> " #secs \
    " seconds). Check code for infinite loops"); \
}

Here’s basically how I’ve implemented this for sub-second thresholds on a Unix OS :

#include "gtest/gtest.h"
#include <unistd.h>
#include <setjmp.h>
#include <signal.h>

static jmp_buf jmp_env;

static void catch_alarm(int sig)
{
    longjmp(jmp_env, 1);
}

#define ASSERT_USECS(fn, usecs) { \
    const auto val = setjmp(jmp_env); \
    if (val == 0) { \
        signal(SIGALRM, catch_alarm); \
        ualarm((usecs), 0); \
        { fn; }; \
        ualarm(0, 0); \
    } else { \
        GTEST_FATAL_FAILURE_(#usecs " usecs timer tripped for " #fn); \
    } }

For example the following should generate a fatal failure message: ASSERT_USECS(usleep(4000), 1000);

And the following should not: ASSERT_USECS(usleep(4000), 8000);

Note that this code has room for improvement. For instance: ualarm is “obsolete”, and any previous signal handling for SIGALRM isn’t restored.