cloudberry: [Bug] Query Causes Segmentation Fault
Apache Cloudberry version
Cloudberry 1.6.0 (this is a pre-Apache release)
What happened
We had a number of segments go down both on 12/16 (multiple waves of segments going down) as well as 12/18 due to a problem query that was run. The log messages show this query along with a segmentation fault error in each segment log that crashed. I’ve obfuscated the table and column names for the sake of privacy in the below error message as well as the provided DDL and problem query, but everything else is exactly the same as it was when it was executed
Segmentation fault","Failed process was running: SELECT col1
, col17
, LAG(col14) OVER (PARTITION BY col1, col17 ORDER BY col1, col17, col14)
, col14
, col14 - LAG(col14) OVER (PARTITION BY col1, col17 ORDER BY col1, col17, col14) AS t
, col14 - LAG(col14) OVER (PARTITION BY col1, col17 ORDER BY col1, col17, col14) < '5 mins' AS v
FROM myschema.mytable
WHERE col14 >= '2024-11-01'
AND col14 < '2024-12-01'
AND col17 = 'FOOBAR’;",,,,,,0,,"postmaster.c",4315,
2024-12-16 22:08:13.008648 UTC,,,p3534838,th-2126882688,,,,0,,,seg171,,,,,"LOG","00000","terminating any other active server processes",,,,,,,0,,"postmaster.c",4023,
2024-12-16 22:08:13.136640 UTC,,,p3534856,th-2126882688,,,,0,,,seg171,,,,,"WARNING","01000","ic-proxy-server: received signal 3",,,,,,,0,,"ic_proxy_main.c",474,
2024-12-16 22:08:13.182293 UTC,,,p3534838,th-2126882688,,,,0,,,seg171,,,,,"LOG","00000","background worker ""ic proxy process"" (PID 3534856) exited with exit code 1",,,,,,,0,,"postmaster.c",4293,
As far as I can tell it doesn’t seem like this query was ever successful, however it’s hard to tell from the logs since they’re pretty verbose. However I did confirm with an engineer that this query was first executed on 12/16 (the day of our segments going down the first time with this query), there’s no entry of it being executed at all on 12/17, and then it was executed again on 12/18 causing the same error and crash. This is a very large table with a lot of partitions and just from taking a quick sampling each partition is around 100GB+ in size, however I can get exact sizes if that’s helpful for analysis.
I’ve attached the core dump analysis files for all 31 segments that went down during this time (this is directly related to issue https://github.com/apache/cloudberry/issues/803) provided by @edespino in his comment here https://github.com/apache/cloudberry/issues/803#issuecomment-2555808199. I’ve also provided the DDL to create the table that I captured using gpbackup and then obfuscated as well as the query itself in files that are attached. Happy to provide any more detail upon request. I’ve tested this in my local Cloudberry docker environment but haven’t been able to recreate the segfault. Mind you though that this test didn’t include any data in the table so I imagine that would influence what steps the query execution plan actually takes.
What you think should happen instead
This query shouldn’t cause a segmentation fault and should just return the results when it’s finished
How to reproduce
I’ve tested this locally and haven’t been able to reproduce the segfault
Operating System
Rocky Linux 8.10 (Green Obsidian)
Anything else
No response
Are you willing to submit PR?
- Yes, I am willing to submit a PR!
Code of Conduct
- I agree to follow this project’s Code of Conduct.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 22 (22 by maintainers)
Just want to give everyone an update, I’ve yet to be able to recreate this in our development environment. I still have a few more things to try and validate though so I’ll keep everyone in the loop
After a thorough investigation of the coredump file, we found that some memory may have been released prematurely, so we have a potential fix: #851. And we are still finding the root cause. @antoniopetrole
@foreyes I’ll take a look at this either later today or tomorrow. We collect and store metrics around memory, disk, and CPU utilization so we should have data on this. Just an FYI I’m in training all of this week so I might not be responding as quickly as normal, thank you again for spending time looking into this!
So I’ve gotten the green light to go ahead and try to recreate this issue in our dev environment. If I manage to do this then the next step will be swapping out the binaries in this environment with a build that has debug flags enabled so we have more information for troubleshooting. I’ll keep everyone in the loop as I go about recreating this
CloudBerry Database Interconnect Crash Analysis
Overview
A widespread interconnect crash occurred across a CloudBerry Database 1.6.0 cluster affecting multiple segments. This analysis examines the crash patterns across the affected segments and hosts.
Would it be possible to share a couple of representative core files from this crash? The analysis shows a consistent pattern:
Having access to two core files would help with:
Core Analysis Script Output and Source
Analysis Output
Analysis Script