ember-light-table: Performance problems

Hey all… sorry once again for the lack of communication.

I finally got around to dedicating an entire evening to tackling this problem from when I last had a couple of months ago (unfortunately, I had to refresh from where I left off). Tonight, I realized that the way I was approaching the problem – drop in replacement & everything working – meant we wouldn’t see any progress on this piece of functionality, and would slow down it’s delivery. So I’ve decided to break it up into more manageable pieces so that we can get the table out into people’s hands as early as possible and then add functionality iteratively.

I’m building out a svelte-table in https://github.com/alexander-alvarez/ember-light-table/tree/vertical-collection that will have the same API as light-table (and will probably end up being exposed through a boolean flag on light-table), and will be built up of components that extend lt-head, lt-body etc and keep the APIs intact as much as possible.

That being said, the goal for my first PR to the repo will be a table that can uses vertical-collection in the body, and works when all columns specify a width. Other functionality will likely still be intact, but not guaranteed to be. I think by doing this as the first milestone we can all see progress on this piece of functionality that people can opt into a more perfomant, most likely less feature-rich component, and we can build it back up to the functionality on par with it’s non performant counterpart.

Please stay tuned, and keep me honest so we can push this forward.

Hey all… sorry for the radio silence. we’ve been working on this, but progress has been slow due to inability to commit cycles to it. Both @offirgolan and I have been pretty slammed. I’m going to try to batch a large amount of work towards it this weekend, so if someone wants to catch up and help out, reaching out after then might be useful.

A few things I’ve learned related to performance with big tables:

API:

• try to return the table data as needed to display. The less transformation you have to do client-side the better.
• use plain arrays rather than relationships

Ember:

• with many hundreds or thousands of records don’t use Ember Data
• the more template driven things are, the slower renders will be. I’ve tried a fully template driven approach for generating column definitions and it didn’t work out that well. Maybe Glimmer 2 will make this a non-issue. I think this addon has a good balance of template driven and class driven approaches though so it might not be impacted as much.
• smoke and mirrors is great, but calculations can get out of whack when measuring table children. If this happens, the side-effect seems to be random whitespace. This is probably why as @SirZach mentioned support for tables might be dropped in the future. It also only works in 1 direction.
• I’ve tried a spike using a smoke-and-mirrors-like approach that works in both directions, using getClientRect within requestAnimationFrame to avoid adding scroll listeners. This works pretty well. The general approach here would be to check each header to determine if it’s still in the viewport. This lets you figure out which columns should render. Rows are a similar concept.
• unbind whatever you are able to using unbound

Ideally the Ember things would be used only when running into issues, I don’t think that they should be a default. Especially considering most apps don’t need to support tables of this size. Maybe we add {{ember-enterprise-table}}. :trollface:

Generally speaking, I’d try and see if there is a different UI design that allows for a more rolled up view of the data, but sometimes you have no choice.

Not sure if any of this helps, but there’s a braindump for you. 😄

Sorry for the wait – https://github.com/offirgolan/ember-light-table/pull/483 Making progress 🛤️ 🛤️ 🚂 🛤️ 🛤️ 🛤️

@offirgolan any update on this? currently using in my app that has 1000+ rows and sadly isn’t working 😢

also, smoke-and-mirrors is absolutely not being abandoned 😛

@billdami I’ve been looking into vertical-collection with @alexander-alvarez but we determined that it was still not fully stable for us to incorporate in this addon. I’ll be experimenting with it in the next couple of weeks and hopefully come up with something.

ftr ember-wormhole currently does not work at all with glimmer2.

Also ftr, I’m close to having a fix in for it.

@offirgolan thank you for merging those fixes so quickly; no worries on not having tested beta. That is what having the project open source is for 😄 @vasilakisfil most certainly.

Even with glimmer2 you will now get a perf boost movin this to smoke and mirrors

@offirgolan FYI, I maintain a helper that allow to check whether a component name resolves to an actual component: https://github.com/xcambar/ember-cli-is-component

It allows :

{{#if (is-component componentName)}}
  do what you have to do
{{/if}}

It could help you working around the component helper only acccepting valid components names.

Table support will be in smoke-and-mirrors 1.0, but not via the main component, it does have it’s own edge cases.

I’ve tried a spike using a smoke-and-mirrors-like approach that works in both directions, using getClientRect within requestAnimationFrame to avoid adding scroll listeners. This works pretty well. The general approach here would be to check each header to determine if it’s still in the viewport. This lets you figure out which columns should render. Rows are a similar concept.

If you look into the virtual-item branch of smoke-and-mirrors, the radar is extractable (and I’ve started to do so at ember-radar https://github.com/runspired/ember-radar). This is essentially an IntersectionObserver polyfill. If @cball want’s to champion helping extract and finalize radar as an addon, very willing to turn over the reigns.

unbind whatever you are able to using unbound

unbound and inline if continue to be two of the best in-template perf boosters there are.

try to return the table data as needed to display. The less transformation you have to do client-side the better. use plain arrays rather than relationships

These are ED issues more than anything else, I have a p.o.c. that shows for large quantities of data we could be roughly 1000x faster with much lower memory overhead than ED is (includes having relationships, and improves the sparse relationship story). These are changes I’m plotting to help bring to ED itself. I’d be willing to clean up the p.o.c. and release it as something like “ember-light-store” or some such.

I would add to this that I’ve consistently found the biggest bottleneck in long lists / large data sets is not render, but the instantiation cost of Ember.Object and subclasses of it. Turning off prototype extensions and converting your data structures to well shaped, simpler classes if you can results in a very significant speed up.

@cball mentioned API considerations that he had to make for ember-spreadsheets to make rows and columns performant.

@cball, would you mind sharing your findings that might be relevant to this?