lorawan-stack: Data views slow down and freeze after receiving many events

Summary

The data views of the console freeze after receiving a lot of events.

Steps to Reproduce

Go to the data view for an entity that receives a lot of events
Wait for some time while observing incoming events (waiting time is relative to the event frequency)
Observe the app becoming slow
Observe the app eventually freezing completely

What do you see now?

Freezing data views

What do you want to see instead?

Data views being stable regardless of the amount of events received

Environment

v3.5.8 and older

How do you propose to implement this?

The problem is likely caused by the redux store being filled considerably when the console is subscribed to an event source. Especially since the packet broker update, it will receive a lot of data per event which easily sucks up memory. My ideas to fix this are:

Introduce a maximum amount of messages that are stored (rest being truncated)
Discard event data for old events
Experiment with storing event data in the browsers localStorage instead of in the redux store. We could then only reference the event data in the redux store. This is assumption-based buy maybe the localStorage can handle large amounts of data better.

@bafonins what do you think?

How do you propose to test this?

Manual testing.

Can you do this yourself and submit a Pull Request?

Yes.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 15 (15 by maintainers)

Most upvoted comments

So this needs to be removed from the current milestone, since it’s not actionable right now as we need a decision on this first.

kschiffer on Mar 16, 2021

This is not in progress anymore. I had to close two attempts to improve this because it was not verifiable that these solutions helped. The situation improved significantly after virtualizing the lists via #3229 but there’s still a point of failure when the event view receives thousands of events. I somehow missed opening a PR that will solve this by limiting the number of events displayed in the console. Will do that now.

kschiffer on Nov 12, 2020

The problem is likely caused by the redux store being filled considerably when the console is subscribed to an event source. Especially since the packet broker update, it will receive a lot of data per event which easily sucks up memory.

Redux store is certainly related here, but the main problem is rendering. After https://github.com/TheThingsNetwork/lorawan-stack/pull/2477#discussion_r421182736 I removed the limit for the number events being rendered in the events view.

Your solution looks fine, but indeed localStorage requires some experimentation

bafonins on Jul 13, 2020