gprofiler: perf segfaults

On Fedora 35 running Kernel 5.16.19-200.fc35.x86_64, gProfiler’s perf segfaults on exeuction.

# setsid ./gprofiler -cu --token '<my_token>' --service-name 'test' &

Yields

[16:34:56] Running gprofiler (version 1.2.19), commandline: '-cu --token lN4td5dwQNA5PyF7D6wYQGstP8Vxh-RYPJIv6ZB0eZA --service-name pmemdev1'
[16:34:56] gProfiler Python version: 3.6.8 (default, Nov 16 2020, 16:55:22)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
[16:34:56] gProfiler deployment mode: standalone_executable
[16:34:56] Kernel uname release: 5.16.19-200.fc35.x86_64
[16:34:56] Kernel uname version: #1 SMP PREEMPT Fri Apr 8 15:34:44 UTC 2022
[16:34:56] Total CPUs: 96
[16:34:56] Total RAM: 250.58 GB
[16:34:56] Linux distribution: Fedora Linux | 35 |
[16:34:56] libc version: glibc-2.34
[16:34:56] Hostname: hostname
[16:34:57] The connection to the server was successfully established (service 'pmemdev1')
[16:35:03] Initialized JavaProfiler (frequency: 11hz, duration: 60s)
[16:35:03] Couldn't create the Java profiler, continuing without it
Traceback (most recent call last):
  File "gprofiler/profilers/factory.py", line 41, in get_profilers
  File "gprofiler/profilers/java.py", line 608, in __init__
  File "gprofiler/profilers/java.py", line 624, in _init_ap_mode
  File "gprofiler/utils/perf.py", line 19, in can_i_use_perf_events
  File "gprofiler/utils/__init__.py", line 265, in run_process
gprofiler.exceptions.CalledProcessError: Command '['/tmp/_MEID3BPp6/gprofiler/resources/perf', 'record', '-o', '/dev/null', '--', '/bin/true']' died with <Signals.SIGSEGV: 11>.
stdout: b''
stderr: b''
[16:35:03] Initialized SystemProfiler (frequency: 11hz, duration: 60s)
[16:35:03] Initialized PythonEbpfProfiler (frequency: 11hz, duration: 60s)
[16:35:05] Initialized RbSpyProfiler (frequency: 11hz, duration: 60s)
[16:35:05] Could not find a Docker daemon or CRI-compatible daemon, profiling data will not include the container names. If you do have a containers runtime and it's not supported, please open a new issue here: https://github.com/Granulate/gprofiler/issues/new
[16:35:05] gProfiler initialized and ready to start profiling
[16:35:05] Starting profiling of Python processes with PyPerf
[16:35:06] Starting perf (fp mode)
[16:35:11] perf failed to start. stdout b'' stderr b''
[16:35:11] Unexpected error occurred
Traceback (most recent call last):
  File "gprofiler/main.py", line 771, in main
  File "gprofiler/main.py", line 357, in run_continuous
  File "gprofiler/main.py", line 148, in __enter__
  File "gprofiler/main.py", line 233, in start
  File "gprofiler/profilers/perf.py", line 192, in start
  File "gprofiler/profilers/perf.py", line 72, in start
  File "gprofiler/utils/__init__.py", line 155, in wait_event
TimeoutError

I see the following in dmesg

# dmesg -T | tail
[Fri Apr 15 16:35:02 2022] perf[13307]: segfault at 10 ip 00007fabc1498b34 sp 00007ffc7b92c5b0 error 4 in libssh.so.4.8.7[7fabc1486000+44000]
[Fri Apr 15 16:35:02 2022] Code: 00 00 00 31 c0 5b c3 b8 ff ff ff ff c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 83 ec 08 48 8d 3d 49 c4 04 00 e8 2c fc fe ff <8b> 80 10 00 00 00 48 83 c4 08 c3 90 f3 0f 1e fa 48 85 ff 74 1b 53

Running the command from gProfiler manually also yields a core dump

# /tmp/_MEID3BPp6/gprofiler/resources/perf record -o /dev/null -- /bin/true
Segmentation fault (core dumped)

Running the included perf does not

# /usr/bin/perf record -o /dev/null -- /bin/true
[ perf record: Woken up 5 times to write data ]
[ perf record: Captured and wrote 0.000 MB /dev/null ]

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 16 (10 by maintainers)

Most upvoted comments

Version 1.2.25 resolves the issue, many thanks!

# setsid ./gprofiler -cu --token='lN4td5dwQNA5PyF7D6wYQGstP8Vxh-RYPJIv6ZB0eZA' --service-name='test'
[08:37:07] Running gProfiler (version 1.2.25), commandline: '-cu --token=... --service-name=test'
[08:37:07] gProfiler arguments: Namespace(app_id_args_filters=[], application_metadata=True, collect_metadata=True, collect_metrics=True, config=None, container_names=True, continuous=True, controller_pid=None, curlify_requests=False, duration=60, flamegraph=True, frequency=11, identify_applications=True, java_async_profiler_args=None, java_async_profiler_buildids=False, java_async_profiler_mcache=30, java_async_profiler_mode='auto', java_async_profiler_safemode=64, java_jattach_timeout=30, java_mode='ap', java_safemode='profiled-oom,profiled-signaled,hserr', java_version_check=True, log_file='/var/log/gprofiler/gprofiler.log', log_rotate_backup_count=1, log_rotate_max_size=5242880, log_to_server=True, log_usage=False, nodejs_mode='disabled', output_dir=None, perf_dwarf_stack_size=8192, perf_inject=False, perf_mode='fp', php_mode='disabled', php_process_filter='php-fpm', pid_ns_check=True, profile_api_version=None, python_add_versions=True, python_mode='auto', python_pyperf_user_stacks_pages=None, rotating_output=False, ruby_mode='rbspy', server_host='https://profiler.granulate.io', server_token='...', server_upload_timeout=120, service_name='test', upload_results=True, verbose=False)
[08:37:07] gProfiler Python version: 3.6.8 (default, Nov 16 2020, 16:55:22)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
[08:37:07] gProfiler deployment mode: standalone_executable
[08:37:07] Kernel uname release: 5.18.0-60.vanilla.1.fc36.x86_64
[08:37:07] Kernel uname version: #1 SMP PREEMPT_DYNAMIC Mon May 23 14:06:04 UTC 2022
[08:37:07] Total CPUs: 96
[08:37:07] Total RAM: 3222.59 GB
[08:37:07] Linux distribution: Fedora Linux | 36 |
[08:37:07] libc version: glibc-2.35
[08:37:07] Hostname: hostname
[08:37:07] The connection to the server was successfully established (service 'test')
[08:37:14] Initialized JavaProfiler (frequency: 11hz, duration: 60s)
[08:37:15] Initialized SystemProfiler (frequency: 11hz, duration: 60s)
[08:37:15] Initialized PythonEbpfProfiler (frequency: 11hz, duration: 60s)
[08:37:17] Initialized RbSpyProfiler (frequency: 11hz, duration: 60s)
[08:37:17] Discovered container runtimes: ['docker']
[08:37:17] gProfiler initialized and ready to start profiling
[08:37:17] Starting profiling of Python processes with PyPerf
[08:37:18] Starting perf (fp mode)
[08:37:18] Started perf (fp mode)
[08:38:21] Successfully uploaded profiling data to the server
[08:39:23] Successfully uploaded profiling data to the server
[08:40:26] Successfully uploaded profiling data to the server
[08:41:28] Successfully uploaded profiling data to the server
[...snip...]

sscargal on Jun 13, 2022