async-lock: Sporadic panic in async-io thread with esp-sys-svc async TLS example
Backtrace: backtrace.txt
Crate versions: Cargo.lock
I am running the example on a ESPcam board, with an esp32 (revision v3.0) Xtensa module (Tinker AI ESP32-S, to be exact).
The example usually crashes once or twice after reset before working.
I’ve only managed to reproduce this with an optimized debug build with codegen-units=1.
[profile.de[v]
debug = true
opt-level = "s"
codegen-units = 1
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 47 (3 by maintainers)
This should finally be resolved in the 1.85.0.0 toolchain.
I’ve tested the repro in the original issue, and with a few modifications I got it to fail on compilers < 1.85.0.0 and it now succeeds on the newest 1.85.0.0 compiler.
We haven’t yet promoted the release to latest, but you can install it manually with
espup install --toolchain-version 1.85.0.0.The LLVM team has merged a change that during our test seemed to have fixed this issue. Hopefully we can include it in the next rustc release for you all to evaluate.
@bugadani I have filled a bug here https://github.com/esp-rs/rust/issues/258 and i have a small program reproducing the issue (at least on my machine) at https://github.com/npmenard/cqueue-deadlock
We have identified an issue with LLVM not upholding alignment requirements, and the LLVM team is working on a fix. I don’t know if this is the root cause for this issue, but I’ll update you with the results after the fix is done.
I know there is little to no progress here, for which I apologize. In the meantime, what I found to work for me is setting the version of
async-ioback to “2.0.0” (as inasync-io = "=2.0.0").We’ll resume the work on finding the root cause shortly.
I am running in the same issue as described by @jothan
I have done some investigation and I believe the issue boils down to the pinning of event_listener In this case the address generated is not 32 bit aligned (ex : 0x3ffe860f not sure if this matters) but more importantly point back to an area of the stack where the first byte is not 0x0. Using GDB i can see that the value it points to is 0x3f (which is the MSB of an address used during the creation of the thread) since the remain bytes are 0 maybe 0xffe8610 would be the correct pinned address? Interestingly if I add a dummy read before pinning like :
The pinned address will be 32bit aligned pointing to a stack area where all bytes are 0 and the bug doesn’t happen.
For reference here are the two assembly dump of the function not working working
Hope that helps @ivmarkov
No idea. Will try to reproduce once I’m back.
I just read about the atomics-in-psram issue. I am using both cores with PSRAM on an ESP32-cam board. I am boxing some futures on the heap that might end up in PSRAM. I think I’m going to revert to a single threaded executor or disable the second core entirely for stability’s sake.
Not sure what is that internal repo, but… I hope you’ll inform us how it is developing 😃
I’ve reported the issue internally
https://doc.rust-lang.org/std/mem/fn.align_of_val.html
The culprit is ConcurrentQueue, it raises the alignment requirement (with a contained
CachePaddedtype, which isrepr(align(64))).Alignment of 64 bytes? 😮 How is that even possible? What might require such an alignment? I think on xtensa, the maximum alignment is 64 bits which is only 8 bytes?
Yes this is aligned to 64bits, but obviously not to 64 bYtes.
So, in my case, the root cause of the crash is:
OnceCellinitializes this struct in an async closure, and taking the value out of a Ready is the pointer read where my app crashes0xsomethingsomethingA0, or0x...30)Now let’s see if I can break a non-idf application with this knowledge 😃
@notgull It still panics in the same spot with debug assertions on (Rust 1.78), however, it does not crash when they are off.