zephyr: Idle Stack Corruption
TL;DR
- A fault occurs when interrupts are frequently used.
- Reason is a corrupted idle stack.
- Timing is critical: changing code or having an active debugger session can affect whether the fault occurs.
- A minimal demo project zephyr-fault-demo reliably reproduces the fault on
same54_xpro.- Only includes a
counterand atimerwith callbacks.
- Only includes a
- The issue has been observed on multiple Zephyr versions, including v4.1.0
- Root cause still undetermined.
Describe the bug
A fault occurs when the counter and the timer are both activated with a callback. It looks, like the idle stack gets corrupted (see Log).
Both the timing seem to be relevant (timing of the two interrupts: systick and counter) and the memory layout in the flash. Adding one nop in the code and the fault does not occur. Also, an active debugging session prevents the fault. This made the bug hard to track down.
I was able to reproduce it with as little code/config as possible. See https://github.com/wag-thom/zephyr-fault-demo for the demo project.
The code has only a counter and a timer, both with a callback. It seems that if bot callbacks occur at the “same” time, the fault happens in the idle-thread.
I still did not figure out what really caused the fault.
I found the bug on the same54_xpro board (cortex m4f, sam0), but I do not know if it is specific to that board/chip.
[Update]: According to @epc-ake also present on esp32s3, see comment.
To Reproduce
- get the code from: https://github.com/wag-thom/zephyr-fault-demo
- init west and update
- build with
west build -p -b same54_xpro -s app - flash and wait for a short time (The debugger has to be detached for the fault to occur. If a debugging session is active, the fault does not occur.)
- attach the debugger to the target
- app is in fault-handler (enable Segger RTT to view logs)
Expected behavior App works without issues.
Impact
App crashes seemingly at random.
High impact
Logs and console output
Note: I filled the whole RAM with 0xAA before flashing.
Fault-Log (with Segger RTT, after attaching to target after a while):
*** Booting Zephyr OS build v4.0.0-79-gc2ad663314b3 ***
[00:00:00.034,000] <err> os: ***** BUS FAULT *****
[00:00:00.034,000] <err> os: Instruction bus error
[00:00:00.034,000] <err> os: r0/a1: 0x20001220 r1/a2: 0x00000000 r2/a3: 0x00000000
[00:00:00.034,000] <err> os: r3/a4: 0x00000000 r12/ip: 0xaaaaaaaa r14/lr: 0x00006951
[00:00:00.034,000] <err> os: xpsr: 0x41000000
[00:00:00.034,000] <err> os: s[ 0]: 0x20000e88 s[ 1]: 0x00000f73 s[ 2]: 0x00000000 s[ 3]: 0xaaaaaaaa
[00:00:00.034,000] <err> os: s[ 4]: 0xaaaaaaaa s[ 5]: 0xaaaaaaaa s[ 6]: 0xaaaaaaaa s[ 7]: 0x20000e88
[00:00:00.034,000] <err> os: s[ 8]: 0xaaaaaaaa s[ 9]: 0xaaaaaaaa s[10]: 0xaaaaaaaa s[11]: 0xaaaaaaaa
[00:00:00.034,000] <err> os: s[12]: 0xaaaaaaaa s[13]: 0xaaaaaaaa s[14]: 0xaaaaaaaa s[15]: 0xaaaaaaaa
[00:00:00.034,000] <err> os: fpscr: 0xaaaaaaaa
[00:00:00.034,000] <err> os: Faulting instruction address (r15/pc): 0x007c5fe6
[00:00:00.034,000] <err> os: >>> ZEPHYR FATAL ERROR 27: Unknown error on CPU 0
[00:00:00.034,000] <err> os: Current thread: 0x20000e88 (idle)
[00:00:00.034,000] <err> os: Halting system
Idle Stack at Fault:
....
AA AA AA AA AA AA AA AA AA AA AA AA AA AA AA AA | AAAAAAAA AAAAAAAA AAAAAAAA AAAAAAAA
20 12 00 20 00 00 00 00 00 00 00 00 00 00 00 00 | 20001220 00000000 00000000 00000000
AA AA AA AA 51 69 00 00 E6 5F 7C 00 00 00 00 41 | AAAAAAAA 00006951 007C5FE6 41000000
88 0E 00 20 73 0F 00 00 00 00 00 00 AA AA AA AA | 20000E88 00000F73 00000000 AAAAAAAA
AA AA AA AA AA AA AA AA AA AA AA AA 88 0E 00 20 | AAAAAAAA AAAAAAAA AAAAAAAA 20000E88
Environment:
- OS: Windows
- Toolchain: Zephyr SDK 0.16.4 *
- Zephyr Version: v4.0.0-79 (c2ad663314b3012f07e04ba083c6d2737a494b5b) *
- Board:
same54_xpro - Code: https://github.com/wag-thom/zephyr-fault-demo
<span>*<span/> [ Update] : Also on latest Zephyr & SDK, see comment.
~With the latest Zephyr v4.1.0, I was not able to reproduce the fault. But because it is so sensitive to timing and memory location, I can’t tell that the issue is not there anymore or the conditions are just not right so it did not occur.
Without “knowing” what caused the fault, I can’t say that it got fixed, just because I did not observe it (yet).~
Additional context
The bug was hard to reliably reproduce. The following seems to have an influence:
- Timing of the
k_timerand thecounter: They are both very fast, so the fault occurs shortly after startup.- I did some testing with GPIOs inside the callbacks, and it seemed, that the fault occurred when both interrupts occurred at the same time.
- slowing down the Timer or the Counter lead to the fault occurring at a later time
- Active debugging session: If a debugging session is active, the fault does not occur (maybe timing related?). I used JLink.
- Flash Memory: for the fault to occur, the flash has to be in a certain place
- Note the
__NOP();at the end of main in the demo-app. If one additional__NOP();is added below, the fault does not occur. If 8 additional__NOP();are added, the fault appears again (8__NOP();= Flash offset of 16 Bytes).- The Logs in that case still have a non-vaild PC-value, but its value is +16 Bytes
- I also added 50k of flash offset (by adding code to main), but the fault was the same. The invalid PC value had the 50k offset.
- If a
__NOP();was added toarch_cpu_idle(arch\arm\core\cortex_m\cpu_idle.c) after the lastISB();command, the Fault changed from an BUS FAULT to a USAGE FAULT (see below for a Log)- If more code is added after the already added
__NOP();in thearch_cpu_idlefunction, it switched back to a BUS FAULT. Adding multiple__NOP();after that did not have any effect on the fault (still occured).
- If more code is added after the already added
- If the code or config is changed, it was sometimes possible to add
__NOP();until the fault occurred. But in other cases, the fault disappeared despite adding >8__NOP();, only to re-appear if something else was changed. This made it hard to make a reliable demo where with as little activated as possible.
- Note the
Things I already did, which had no effect on the Fault:
- doubled all stack sizes
- offset the RAM
- the following CONFIG options also had no effect if on or off, so I turned as much as I could off in the demo
- ARM_MPU, HW_STACK_PROTECTION, TIMESLICING, GPIO, SERIAL, CONSOLE, ASSERT
- IRQ NVIC priority of SysTick (timer) and the hardware timer tc32 (counter)
Log when USAGE FAULT occurred:
...
SLEEP_IF_ALLOWED(__WFI);
__enable_irq();
__ISB();
__NOP(); // <--- added
}
Fault-Log (with Segger RTT, after attaching to target after a while):
*** Booting Zephyr OS build v4.0.0-79-gc2ad663314b3 ***
[00:00:00.030,000] <err> os: ***** USAGE FAULT *****
[00:00:00.030,000] <err> os: Attempt to execute undefined instruction
[00:00:00.030,000] <err> os: r0/a1: 0x20001220 r1/a2: 0x00000000 r2/a3: 0x00000000
[00:00:00.030,000] <err> os: r3/a4: 0x00000000 r12/ip: 0xaaaaaaaa r14/lr: 0x00006953
[00:00:00.030,000] <err> os: xpsr: 0x41000000
[00:00:00.030,000] <err> os: s[ 0]: 0x20000e88 s[ 1]: 0x00000f73 s[ 2]: 0x00000000 s[ 3]: 0xaaaaaaaa
[00:00:00.030,000] <err> os: s[ 4]: 0xaaaaaaaa s[ 5]: 0xaaaaaaaa s[ 6]: 0xaaaaaaaa s[ 7]: 0x20000e88
[00:00:00.030,000] <err> os: s[ 8]: 0xaaaaaaaa s[ 9]: 0xaaaaaaaa s[10]: 0xaaaaaaaa s[11]: 0xaaaaaaaa
[00:00:00.030,000] <err> os: s[12]: 0xaaaaaaaa s[13]: 0xaaaaaaaa s[14]: 0xaaaaaaaa s[15]: 0xaaaaaaaa
[00:00:00.030,000] <err> os: fpscr: 0xaaaaaaaa
[00:00:00.030,000] <err> os: Faulting instruction address (r15/pc): 0x000065d2
[00:00:00.030,000] <err> os: >>> ZEPHYR FATAL ERROR 36: Unknown error on CPU 0
[00:00:00.030,000] <err> os: Current thread: 0x20000e88 (idle)
[00:00:00.030,000] <err> os: Halting system
Idle Stack at Fault:
....
AA AA AA AA AA AA AA AA AA AA AA AA AA AA AA AA | AAAAAAAA AAAAAAAA AAAAAAAA AAAAAAAA
20 12 00 20 00 00 00 00 00 00 00 00 00 00 00 00 | 20001220 00000000 00000000 00000000
AA AA AA AA 53 69 00 00 D2 65 00 00 00 00 00 41 | AAAAAAAA 00006953 000065D2 41000000
88 0E 00 20 73 0F 00 00 00 00 00 00 AA AA AA AA | 20000E88 00000F73 00000000 AAAAAAAA
AA AA AA AA AA AA AA AA AA AA AA AA 88 0E 00 20 | AAAAAAAA AAAAAAAA AAAAAAAA 20000E88
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 2
- Comments: 16 (10 by maintainers)
@nashif If this Bug is present on multiple platforms and causes a crash if many interrupts occur (which should be quite common use case), that would have pretty high impact on Zephyr. Could you re-evaluate your priority assignment?
I encounter a similar issue on our esp32s3 based project. We use BT, I2C and Cam peripherals and many interrupts get raised frequently. If I disable just one of the peripherals the application works without any issue. If all three of them are enabled I randomly run into a corrupted idle stack exception. I’m also using Zepyhr (4.1.0) and SDK (0.17.0) versions but also tried it on newest main.
[update] After I reduced the traffic on the I2C interface, the issue is occurring way less frequently (e.g after around 20min instead of 3min)
Hi @andyross ,
I think we need your wisdom. It seems that @kowalewskijan could trace the problem. Do you have any tip in regards to this problem ?