pants: tests/python/pants_test/base:exception_sink_integration is flaky
When run locally, this completes relatively quickly. But in some number of runs, it seems to hang forever, triggering a 360 second test timeout in travis.
tests/python/pants_test/base:exception_sink_integration .....Command '['/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/bin/python3.6', '/Users/travis/build/pantsbuild/pants/.pants.d/test/pytest-prep/CPython-3.6.5/929c23cae3b600b495a5d319ae6c47e8b41a2667', '-c', '/dev/null', '-ocache_dir=/Users/travis/build/pantsbuild/pants/.pants.d/test/pytest/.pytest_cache', '--junitxml', '/Users/travis/build/pantsbuild/pants/.pants.d/test/pytest/tests.python.pants_test.base.exception_sink_integration/junitxml/TEST-tests.python.pants_test.base.exception_sink_integration.xml', '--confcutdir', '/Users/travis/build/pantsbuild/pants', '--continue-on-collection-errors', '--color', 'yes', '-q', '-rfa', '--rootdir', '/Users/travis/build/pantsbuild/pants', '-p', '__pants_backend_python_tasks_pytest_prep_pytest_plugin__', '--pants-sources-map-path', '/Users/travis/build/pantsbuild/pants/.pants.d/test/pytest/tmpg79dyv4r/sources_map.json', '/Users/travis/build/pantsbuild/pants/.pants.d/pyprep/sources/49f3f1d9d9dc377d027f9fb364db7fffbb6a5ab9/pants_test/base/test_exception_sink_integration.py']' timed out after 360 seconds
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 23 (23 by maintainers)
Commits related to this issue
- Skip two flaky tests and tweak more test timeouts (#8687) ### Skip some exception sink integration tests on macOS These shards have chronically flaked by hanging since at least July https://github.c... — committed to pantsbuild/pants by Eric-Arellano 5 years ago
- Port the pantsd nailgun server to rust (#9722) ### Problem The setup and teardown of each request made to the nailgun server in `pantsd` had become quite complicated over time... and consequently,... — committed to pantsbuild/pants by deleted user 4 years ago
- Disable capturing core dumps in travis. (#9769) ### Problem A while back we started capturing core dumps "globally" in travis. But in practice we have never consumed them, and I'm fairly certain t... — committed to pantsbuild/pants by deleted user 4 years ago
- Disable capturing core dumps in travis. (#9769) ### Problem A while back we started capturing core dumps "globally" in travis. But in practice we have never consumed them, and I'm fairly certain t... — committed to pantsbuild/pants by deleted user 4 years ago
Seen again in master.
This is probably our highest priority flaky test, as it seems to just hang fairly frequently.
I’m looking into this today. I agree with Stu that this is likely our highest priority flake.
Locally, I ran a script to repeat until failure. First run, it took 71 attempts. Second run, it took 131 attempts to fail. This translates to 1.3% of runs failing and 0.7% of runs failing, respectively. In CI, it seems the number is closer to 20%. I’m going to try debugging in CI instead.