cocotb: Simple Python exception causes segmentation fault on vcs
We have been struggling with random segmentation faults with our verification suite and VCS and until now we had assumed it was a problem with our code. However, recently I have noticed similar behaviour with a very simple test case which appears to be a problem with cocotb itself.
Let’s say I have the following simple test:
import cocotb
@cocotb.test()
def check_top_level(dut):
"""Checks the simulation was built and can be clocked."""
yield do_something(dut)
Where in another file I have this:
import cocotb
# Note I have commented out this import on purpose...
# from cocotb.triggers import Timer
@cocotb.coroutine
def simple_coroutine(dut):
yield Timer(500)
@cocotb.coroutine
def do_something(dut):
# dut.i_bank_rst__b_a = 0 # Uncomment me to cause a segfault
yield simple_coroutine(dut)
Now, if you run this as-is you will get an exception due to Timer not being resolved. However, if you access any of the verilog itself (e.g uncomment the line that resets a reset signal in our design), then what would normally be thrown as an exception now causes a segfault. This seems to be an issue with the embedded Python interpreter shutting down before the exception is printed and then Python tries to access invalid memory and causes a segfault (at least that is our guess).
We have seen similar issues (segfault or a lockup) happen when a Python exception happens after we have accessed the verilog, but before the first yield of a coroutine. It seems to be related to the embedded Python in cocotb doing something wrong when shutting down.
I think if we could get to the root of this issue in cocotb, we could then begin to get to the real root of our segfaults - which could be a simple Python exception coming from somewhere.
About this issue
- Original URL
- State: open
- Created 5 years ago
- Reactions: 1
- Comments: 45 (19 by maintainers)
I was hitting the same issue with VCS and can confirm that either of the following fixes the issue if put at the end of the sequence:
Icarus and Aldec Active-HDL return from
gpi_sim_end, which is why I added thesim_endingflag insimulatormodule.cand set it instop_simulator. Once the cocotb scheduler returns back intohandle_gpi_callback,gpi_sim_endis called which eventually callsPy_Finalize.@markmelvin The line in question should not need to be conditioned. It’s worth investigating why it is in your case, and fix that issue instead.
Indeed, running my original test case from this issue, I can now see a proper shutdown and error message with
registered_impls[0]->sim_end();commented out as per above. I will try to step through the code and see what exactly is going wrong here and post more info as I get it.@markmelvin: if you’re desperate to work out what your exception is, add some logging to
cocotb.outcomes.Error.__init__in your local installation, which will definitely be reached if any error occursSetting
DEBUG=1gives a bit more info for this particular issue: