polymerfire: firebase-query is very slow when loading an array with more than just a few items

Description

I am loading a firebase array with around 12000 items, using a Polymer element like:

<link rel="import" href="/bower_components/polymer/polymer.html">

<link rel="import" href="/bower_components/polymerfire/firebase-query.html">

<dom-module id="loader">
  <template>
    <firebase-query id="query"
                    path="/data"
                    data="{{data}}">
    </firebase-query>
  </template>

  <script>
    (function () {
      'use strict';

      Polymer ({
        is: 'loader',

        properties: {
          data: {
            type: Object,
            value: [],
            notify: true
          }
        },

        ready: function () {
          var self = this;

          setTimeout (function () {
            console.log ('Data load took ' + (self.$.query.__loadEnd - self.$.query.__loadStart) + 'ms');
          }, 60000);
        }
      });
    }) ();
  </script>
</dom-module>

The setTimeout () callback is for my instrumentation, which in the current version of firebase-query looks like:

--- firebase-query.html.orig	2017-04-28 12:13:59.221771214 +0100
+++ firebase-query.html.instr	2017-04-28 14:31:49.199146378 +0100
@@ -308,6 +308,10 @@
         },
 
         __onFirebaseChildAdded: function(snapshot, previousChildKey) {
+          if (!this.__loadStart) {
+            this.__loadStart = performance.now ();
+          }
+
           var key = snapshot.key;
           var value = snapshot.val();
           var previousChildIndex = this.__indexFromKey(previousChildKey);
@@ -318,6 +322,7 @@
 
           this.__map[key] = value;
           this.splice('data', previousChildIndex + 1, 0, value);
+          this.__loadEnd = performance.now ();
         },
 
         __onFirebaseChildRemoved: function(snapshot) {

Expected outcome

My app should load the data quickly, without interrupting the user experience too severely.

Actual outcome

My instrumentation timings for several loads are (times in ms):

37324.61 37408.37 36633.60 37129.21 38887.08

Yes, that is nearly 40 seconds to load just 12000 items! Worse, the browser is completely frozen while this is happening.

Steps to reproduce

  1. Have a decent-sized array in Firebase
  2. Load it as above
  3. Try to interact with page elements while the load happens

Browsers Affected

  • Chrome
  • Firefox
  • Safari 9
  • Safari 8
  • Safari 7
  • Edge
  • IE 11
  • IE 10

I’ve tested with Chrome, but I doubt this is browser specific.

About this issue

  • Original URL
  • State: open
  • Created 7 years ago
  • Reactions: 1
  • Comments: 34 (15 by maintainers)

Commits related to this issue

Most upvoted comments

The Firebase SDK definitely caches locally in-memory, so an on(‘value’) will start a listener and you can separately do on(‘child_added’). This will not incur double network cost.

On Fri, May 5, 2017 at 11:55 AM Toni-Jan Keith Monserrat < notifications@github.com> wrote:

@dickp https://github.com/dickp Ok, I’ll try to look it up on your side and do a snapshot of the network. As far as I checked, the child_added event already serves the 12000 items in 1.2 seconds. Will try it out if the changes you made are spot on

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/firebase/polymerfire/issues/212#issuecomment-299547667, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAD_gSQk_KmaTfqQpQof-Gtm1Zn7L8Uks5r23CkgaJpZM4NLvYA .

I don’t really think you should be using firebase-query for anything larger than a couple hundred items.

It would be nice to see the limitations documented then, before anyone else runs into this.

@mikelehen thank you for the clarification regarding the value event. In that case, I think that some minor changes to @dickp 's design could be sufficient to improve performance in a general way.

Also the child-added handler invokes Polymer data-binding path notifications for the array each time.

I agree with @mbleigh 's assessment that the number of items being considered is unrealistic. Even with the speed up, there will be problems rendering this much data in the client for most non-trivial usages.

That said, seems 40s is pretty bad. It seems likely that the problem is at a higher level than the Firebase client library. However, it would be nice to confirm that with profiling.

If all 12000 items has a size of 2MB, you are downloading another 2MB because of the child_added listener. That is I think a no-no in web performance.

The timings I posted at the top of this thread would beg to differ. 40 seconds using child_added on its own, including 10s of seconds of frozen browser UI, vs with value it’s 40 milliseconds to the last child_added event even if the data is loaded twice.

Whatever. The patch works for me, take it or leave it. If you don’t want a 1000x speed improvement it makes no difference to me.