datafusion-comet: Spark executor fail to start occasionally with SIGILL
Describe the bug
Hi,
We experience occasional failures of Spark executors with following
│ # │
│ # SIGILL (0x4) at pc=0x00007f68096df4f8, pid=18, tid=0x00007f68121ff640 │
│ # │
│ # JRE version: OpenJDK Runtime Environment (Zulu 8.74.0.17-CA-linux64) (8.0_392-b08) (build 1.8.0_392-b08) │
│ # Java VM: OpenJDK 64-Bit Server VM (25.392-b08 mixed mode linux-amd64 compressed oops) │
│ # Problematic frame: │
│ # C [libcomet-9087565472916431370.so+0x30df4f8] core::hash::BuildHasher::hash_one::he592d13170e1e1dc+0x18 │
│ # │
│ # Core dump written. Default location: /opt/spark/work-dir/core or core.18 │
│ # │
│ # An error report file with more information is saved as: │
│ # /opt/spark/work-dir/hs_err_pid18.log │
│ # │
│ # If you would like to submit a bug report, please visit: │
│ # http://www.azul.com/support/ │
│ # The crash happened outside the Java Virtual Machine in native code. │
│ # See problematic frame for where to report the bug.
Our env:
Spark 3.5.4 - Comet version 0.7.0
Linux qa-michael-tarano-pr-84194-5432-b5d2bb95f6539772-exec-40 6.1.123-1k8s.el8.x86_64 #1 SMP
PREEMPT_DYNAMIC Mon Jan 13 18:17:08 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Unfortunately I wasn’t able to grab a dump as pods weren’t available 😦
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (8 by maintainers)
After testing on much heavier input/shuffle/output (200GB/2TB/100GB) the performance boost was amazing, 2x at least. Thanks for help.
Just
-Ctarget-cpu=broadwell. Sorry I should have been more specific.