datafusion-comet: Spark executor fail to start occasionally with SIGILL

Describe the bug

Hi,

We experience occasional failures of Spark executors with following

│ #                                                                                                                                                                                                                                                                                       │
│ #  SIGILL (0x4) at pc=0x00007f68096df4f8, pid=18, tid=0x00007f68121ff640                                                                                                                                                                                                                │
│ #                                                                                                                                                                                                                                                                                       │
│ # JRE version: OpenJDK Runtime Environment (Zulu 8.74.0.17-CA-linux64) (8.0_392-b08) (build 1.8.0_392-b08)                                                                                                                                                                              │
│ # Java VM: OpenJDK 64-Bit Server VM (25.392-b08 mixed mode linux-amd64 compressed oops)                                                                                                                                                                                                 │
│ # Problematic frame:                                                                                                                                                                                                                                                                    │
│ # C  [libcomet-9087565472916431370.so+0x30df4f8]  core::hash::BuildHasher::hash_one::he592d13170e1e1dc+0x18                                                                                                                                                                             │
│ #                                                                                                                                                                                                                                                                                       │
│ # Core dump written. Default location: /opt/spark/work-dir/core or core.18                                                                                                                                                                                                              │
│ #                                                                                                                                                                                                                                                                                       │
│ # An error report file with more information is saved as:                                                                                                                                                                                                                               │
│ # /opt/spark/work-dir/hs_err_pid18.log                                                                                                                                                                                                                                                  │
│ #                                                                                                                                                                                                                                                                                       │
│ # If you would like to submit a bug report, please visit:                                                                                                                                                                                                                               │
│ #   http://www.azul.com/support/                                                                                                                                                                                                                                                        │
│ # The crash happened outside the Java Virtual Machine in native code.                                                                                                                                                                                                                   │
│ # See problematic frame for where to report the bug. 

Our env:

Spark 3.5.4 - Comet version 0.7.0

Linux qa-michael-tarano-pr-84194-5432-b5d2bb95f6539772-exec-40 6.1.123-1k8s.el8.x86_64 #1 SMP 
PREEMPT_DYNAMIC Mon Jan 13 18:17:08 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Unfortunately I wasn’t able to grab a dump as pods weren’t available 😦

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (8 by maintainers)

Most upvoted comments

Unfortunately can’t see any visible gains of working with comet vs bare Spark 😕

Were you able to confirm that queries were running in Comet?

After testing on much heavier input/shuffle/output (200GB/2TB/100GB) the performance boost was amazing, 2x at least. Thanks for help.

Just -Ctarget-cpu=broadwell. Sorry I should have been more specific.