async-lock: Sporadic panic in async-io thread with esp-sys-svc async TLS example

Code: https://github.com/esp-rs/esp-idf-svc/blob/4fff46bba1be66ae3c6945d3b8bda30a589f6f6b/examples/tls_async.rs

Backtrace: backtrace.txt

Crate versions: Cargo.lock

I am running the example on a ESPcam board, with an esp32 (revision v3.0) Xtensa module (Tinker AI ESP32-S, to be exact).

The example usually crashes once or twice after reset before working.

I’ve only managed to reproduce this with an optimized debug build with codegen-units=1.

[profile.de[v]
debug = true
opt-level = "s"
codegen-units = 1

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 47 (3 by maintainers)

Most upvoted comments

This should finally be resolved in the 1.85.0.0 toolchain.

I’ve tested the repro in the original issue, and with a few modifications I got it to fail on compilers < 1.85.0.0 and it now succeeds on the newest 1.85.0.0 compiler.

We haven’t yet promoted the release to latest, but you can install it manually with espup install --toolchain-version 1.85.0.0.

The LLVM team has merged a change that during our test seemed to have fixed this issue. Hopefully we can include it in the next rustc release for you all to evaluate.

@bugadani I have filled a bug here https://github.com/esp-rs/rust/issues/258 and i have a small program reproducing the issue (at least on my machine) at https://github.com/npmenard/cqueue-deadlock

We have identified an issue with LLVM not upholding alignment requirements, and the LLVM team is working on a fix. I don’t know if this is the root cause for this issue, but I’ll update you with the results after the fix is done.

I know there is little to no progress here, for which I apologize. In the meantime, what I found to work for me is setting the version of async-io back to “2.0.0” (as in async-io = "=2.0.0").

We’ll resume the work on finding the root cause shortly.

I am running in the same issue as described by @jothan

I have done some investigation and I believe the issue boils down to the pinning of event_listener In this case the address generated is not 32 bit aligned (ex : 0x3ffe860f not sure if this matters) but more importantly point back to an area of the stack where the first byte is not 0x0. Using GDB i can see that the value it points to is 0x3f (which is the MSB of an address used during the creation of the thread) since the remain bytes are 0 maybe 0xffe8610 would be the correct pinned address? Interestingly if I add a dummy read before pinning like :

let event_listener = EventListener::new();
let _ = event_listener.is_listening();
pin!(event_listener);

The pinned address will be 32bit aligned pointing to a stack area where all bytes are 0 and the bug doesn’t happen.

For reference here are the two assembly dump of the function not working working

Hope that helps @ivmarkov

cc @ivmarkov Any idea why this would be happening?

No idea. Will try to reproduce once I’m back.

I just read about the atomics-in-psram issue. I am using both cores with PSRAM on an ESP32-cam board. I am boxing some futures on the heap that might end up in PSRAM. I think I’m going to revert to a single threaded executor or disable the second core entirely for stability’s sake.

Not sure what is that internal repo, but… I hope you’ll inform us how it is developing 😃

I’ve reported the issue internally

https://doc.rust-lang.org/std/mem/fn.align_of_val.html

Returns the ABI-required minimum alignment of the type of the value that val points to in bytes.

The culprit is ConcurrentQueue, it raises the alignment requirement (with a contained CachePadded type, which is repr(align(64))).

So, in my case, the root cause of the crash is:

  • Reactor is a 448 byte struct, with alignment 64

Alignment of 64 bytes? 😮 How is that even possible? What might require such an alignment? I think on xtensa, the maximum alignment is 64 bits which is only 8 bytes?

  • OnceCell initializes this struct in an async closure, and taking the value out of a Ready is the pointer read where my app crashes
  • The cause of the crash is that Ready and its contents aren’t placed on a 64-byte allocated address. (in my case, 0xsomethingsomethingA0, or 0x...30)

Yes this is aligned to 64bits, but obviously not to 64 bYtes.

So, in my case, the root cause of the crash is:

  • Reactor is a 448 byte struct, with alignment 64
  • OnceCell initializes this struct in an async closure, and taking the value out of a Ready is the pointer read where my app crashes
  • The cause of the crash is that Ready and its contents aren’t placed on a 64-byte allocated address. (in my case, 0xsomethingsomethingA0, or 0x...30)

Now let’s see if I can break a non-idf application with this knowledge 😃

@notgull It still panics in the same spot with debug assertions on (Rust 1.78), however, it does not crash when they are off.