WLED: LED data corruption with RMT driver
What happened?
After upgrading a QuinLED-Dig-Quad to WLED 0.15.0, LED segments on GPIOs 1 and 16 experience significant noise and flashing. An LED segment on GPIO 3 operates fine without any issue.
Segment 1 GPIO 3 - 629 LEDs Segment 2 GPIO 1 - 161 LEDs Segment 3 GPIO 16 - 171 LEDs
This particular setup has been running flawlessly for just over 3 years. The issue was immediately resolved by downgrading to WLED 0.14.3.
To Reproduce Bug
- Upgrade a QuinLED-Dig-Quad to WLED 0.15.0
- Activate any preset or animation
- Flashing/Noise is prominent on LED segments on GPIO 1 and 16
Expected Behavior
- No noise/flashing on any LED segment
Install Method
Binary from WLED.me
What version of WLED?
WLED 0.15.0 2412100
Which microcontroller/board are you seeing the problem on?
ESP32
Relevant log/trace output
I don't have any logs at the moment, I have since downgraded to WLED 0.14.3. However, I would be more than happy to re-upgrade and capture any logs/traces/configs that would aid in troubleshooting.
Anything else?
This appears to be similar to report #3976. That was submitted for 0.15.0b2 and not the full release however.
Code of Conduct
- I agree to follow this project’s Code of Conduct
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 8
- Comments: 69 (13 by maintainers)
I’m happy to report that the high-priority interrupt driver is working well on my first test board - I can now get stable output with any amount of network traffic. I’m working on making sure it builds and operates on more chips. With any luck I should be able to pass along a beta on the weekend.
@hacustom why are you regressing? You already reported that 6+ outputs resolves the issue.
It is now apparent that the root cause is RMT driver (and not I2S as I thought on a first glance after a revisit).
@willmmiles I suggest you don’t spend a lot of time debugging RMT driver as it is far easier to (always) use parallel I2S. Especially once @Makuna manages to reduce backend buffers (which will allow larger LED count without depleting RAM).
@Makuna I’ve implemented complete pixel buffer management in/for 0.16 (to allow layer blending) and we no longer rely on NPB to provide
GetPixelColor()or maintain double buffers. So it might be possible to instruct NPB to use single buffer (either during creation orbegin()or even at compile time) if you ever want to explore that (I have not looked at new additions to NPB in quite some time so please excuse me if that is already implemented).For anyone experiencing this issue with any version of 0.15, please revert to 0.14 and wait for 0.16 or create additional phantom outputs (1 pixel is enough) while keeping other outputs with less than 300 pixels. I don’t think we’ll backport RMT solution into 0.15 as it is already waiting in beta for quite some time without any proper feedback from beta testers and 0.16 will bring so many enhancements and performance improvements that it makes no sense to invest time in 0.15.x.
I have a fix for that ready.
@MakerMatrix if the port would be simple I would agree with you, but it is not (at least the way I see it ATM). However you are welcome to join the efforts.
@willmmiles we should be a few (couple of?) months away from releasing 0.16 so IMO we should focus on that and try to expedite it instead. Not everyone is hitting RMT bug (in fact only users with 2 to 5 outputs may be effected by it and not all of them will see it). But I understand the desire to hunt down the nasty bug.
@Makuna my parallel I2S tests were performed in April/May 2024 (NPB versions 2.7.9 and 2.8.0). I am currently using 2.8.3 in my development fork but did not do any memory testing on it. IIRC the memory calculation formula was
channels * max_pixels_per_channel * 8 * 3where 8 comes from parallel I2S and 3 comes from three buffers (front/back/transmit) but I can’t find my notes any more. However, if memory serves me well, if I used more than 300 pixels memory on an ESP32 got so low that it would crash due to OOM exception during WiFi activity which doubles the above formula. I know you were working on some modifications to backend (coreshader?) but didn’t follow due to the lack of time. Looking back at commit history I favoured I2S after the switch from 2.7.5 to 2.7.8 and this might have been due to user reports of flashing glitches.@hacustom I was explaining a workaround anyone stuck with (or needing) 0.15 may want to try. There is no need to modify “segments” as you wrongly explained (but deleted the post). If you are selling products to your customers that use WLED then you should fork and only use your fork’s binaries and keep that locked. Or invest time and help develop a correct solution.
I’m not satisifed with “just don’t use RMT”; I’d like to make sure I fully understand the depth and scope of the issue, and that’s not a solution for anyone with longer output lengths. This used to work on 0.14 which was not so long ago - what did we change that triggered this regression?
I’ve been doing a survey of “what do other projects do”; it seems like our friends over at MoonModules had, at some point, observed some trouble with I2S – enough that they’ve got a compile-time flag to set the default assignment to RMT first. It looks like FastLED actually implements “detect and abort” logic if there’s an RMT interrupt timing issue. And of course there’s an entirely new RMT driver scheme provided by IDF v5 to support the improved capabilities of new chips – it looks like Espressif has added DMA hardware for the RMTs in their newer chip designs expressly to address problems with interrupt latency for our kind of use case.
As for backporting to 0.15.1: I believe that this is a big enough problem to warrant a real fix, even if we’re just changing the default hardware assignment to parallel I2S as a mitigation. Whatever it turns out to be, I don’t expect it to be a difficult port, and I think we’d be doing users a big disservice if we ship something this broken as the next release.
Long at the short of it: I’m still investigating and I’ll hopefully have more answers as to the root cause and possible mitigation soon.
@blazoncek Right, let me walk it all back. The problem I’m seeing is with the RMT driver – I had the test inverted in my debug code. Parallel I2S is rock solid by comparison. 😃 (I do think there is a possible weakness to interrupt latency in NPB’s I2S DMA logic; but this doesn’t seem to be a problem in practice.)
The glitches I am seeing are induced by network traffic, and look to me like a FIFO underrun: at some point partway through the output, the data “slips” one (or more) bit times, resulting in shifted or corrupted colors. Tellingly, the glitch events can cause LEDs past the end of the configured count to light.
I’ve reproduced this on a couple of different ESP32 (WROOM/WROVER) boards from different manufacturers.
I’m digging in to the RMT driver now.
I’m able to replicate this on my test system, even with the latest 0.16 build. There seems to be an issue with the Parallel I2S hardware driver which results in occasional output timing errors during web server accesses. I am investigating.
@MakerMatrix yes it was explained (elsewhere). GPIO don’t matter but number of LEDs do. If you use more than 300 LEDs per output, WLED will use RMT peripheral for output instead of I2S (behaviour from 0.14).
I believe it’s fixed in v0.15.1-beta2, could you give that one a try @MakerMatrix ?
If needed I can build a quick test setup and can troubleshoot on Discord too, apologies this made it through the beta somehow, I had not noticed in on any testing setups I had before. Let me know if you need anything!
First test builds (based on 0.15.x branch) are up: https://github.com/willmmiles/WLED/actions/runs/16158643608 Use the “rmt” bins; I don’t know why the CI system also zipped up older files along with them. This should be considered quite experimental; please don’t install them on a device that you can’t easily reflash via USB.
So far I’ve tested ESP32 (IDF v3 and v4), and ESP32-S2. Good chance S3 will also work. C3 is not supported yet. I’ve now received an S3 and C3 board for my bench but I don’t have them set up yet.
I haven’t got a thorough analysis on all platforms - still building out that hardware test matrix.
What I know is:
Testing is still pending on:
I should have more results over the weekend - most of the boards have arrived, I still need to assemble the test jig.
Known safe configurations:
Or to sum up: don’t use RMT. (This is, of course, no help to anyone who’s got two strips with different timings connected to one controller.)
Sorry I don’t have full information yet - I’ve been focusing more on a fix than building the test matrix.
A quick update:
I would tend to agree - or as I said above, at least provide some documentation and a clean way to see and control what bus gets used for which outputs. The hacky and unclear workaround of a known bug shouldn’t be the only option in production code. I lost many hours to it.
A good start would be to report which outputs are programmed and what kind of bus they use, under (i)->Instances for each instance. That might be a great boon to users and devs alike for troubleshooting issues.
Nice chat (revealing). TL:DR you may want to force parallel I2S which is triggered by using more than 5 outputs. Remember, they need to be less than 300 pixels in length.
When you have >5 outputs WLED will switch to parallel I2S for firs 8 outputs and use RMT for last 8 (if configured). Unfortunately parallel I2S uses a lot of buffers so we needed to enforce a limit of 300 pixels to prevent memory drain.
As far as GPIO go, all IO are equal (ESP32 uses GPIO muxer).
The first defined LED output will use the I2S driver (ie. the one at the top of the LED configuration page). Subsequent outputs will use the RMT driver instead. If you remap the pins between the different outputs, I would expect that you see the first one is consistently stable, regardless of what pin it is assigned to, and the others glitchy.
@willmmiles according to @Makuna experiment with cadence 4 for parallel I2S prior to do any other investigation. Cadence 3 and cadence 4 have slightly different timings which may behave differently on edge cases (out of specs systems, plenty of IRQs, etc). Changing cadence requires compile override.
@MakerMatrix change to different hardware (controller board and ESP) to rule out HW/SW issue.
Out of my 10+ production units, none exhibits any visual glitch. One or many outputs, <100 LEDs and >500 LEDs per output. I use variety of HW form @intermittech, @srg74, AthomTech and a few other OS boards, including some soldered at home. My WiFi network contains over 100 units, constantly chatting.
Interesting, from my understanding the main issue was fixed in v0.15.1-beta2 so I’m not 100% sure what’s going on, I will give it a try sometime soon (currently in a crunch to get a video out)!
IMHO the dig-quad is the best hardware for the task at hand. I believe it’s important that WLED give users more control over when and if the new rendering peripheral is engaged since in my studies over the past few days it is giving a lot of people headaches, not only QuinLED customers (for example #4211 mentioned above). I tried the suggested workaround to make the last LED entry >300 on 0.15.0 but either didn’t understand that entirely, or it didn’t work.
For now everything is happy on 0.14.4-Soundreactive but I’d like to not get orphaned beyond that release. I say this as someone who has donated to the project, not just a total freeloader. 😉
In any case the release notes indicate that it’s fixed, somehow, in 0.16-alpha so until then…
@JorinL PWM is unrelated to digital flickering. Please open a new issue.
Hey there, I have the same problem on my 3 PWM strips. When I cut the transitiontime it is gone - with transition it flashes when turning on before dimming up and when turning off while dimming down a flash and dimming further down.
I added a small video of the current situation. https://github.com/user-attachments/assets/f6009672-1853-451c-b2fd-f404a0fe5acc
The only real change with RMT between those versions is the ability to compile using IDF5. This was just some minor headers include changes to react to the IDF change but still outputs warnings about deprecated API use. Did you turn on ESP32 core debug errors/warnings/info level to see if anything shows up in the debug output?
I’m dealing with this same issue, except it’s on a single ESP32. There are 260 LEDs in this configuration, all of which are in GPIO 16.
it’s only the last section where I’m adding power back in after a split and it’s after that point that the LEDs are misfiring. I’ve posted video of it flashing blue and red after a few minutes in the discord channel. It was working fine in 14.4. I’m not sure if I’m sending too much power and the additional power is fucking with it or something. Let me know if there’s anything you need me to do.
Please test a set-up with at least 6 outputs (WS281x). You do not need to have actual LEDs connected or GPIOs attached. It is important to have GPIO 1, 3 & 16 as first 3 outputs and each output must not exceed 300 LEDs
Periodic issue leads me to believe this may be network related.