cypress: Bad XHR performance with Google Cloud Firestore. Like really bad.

I believe I’ve partially worked through this using the comment above by @andrew-teirney. That comment didn’t exactly do the trick for me, but it was close, and it led me to greatly increase my understanding of how Cypress is working. And with it I was able to comment out the experimentalForceLongPolling:true option that was causing Cypress to be so slow and unreliable for us while using Firebase/Firestore. I’ll try to explain…

This is the code I put in the cypress/plugins/index.js file:

on('before:browser:launch', (browser = {}, launchOptions) => {
    if (browser.name === 'chrome') {
      // Add localhost:8080, the Firestore emulator host:port when running locally, to the Chrome proxy bypass
      // So Cypress doesn't jack with it
      launchOptions.args.push('--proxy-bypass-list=<-loopback>,localhost:8080');
      /*
      // Options that provide marginal memory improvement in Chrome, but seem to break headed mode
      // Leaving them out for now as I don't fully understand them and the improvement is marginal
      launchOptions.args.push('--ChromeOSMemoryPressureHandling');
      launchOptions.args.push('--renderer-process-limit=1');
      launchOptions.args.push('--single-process');
      launchOptions.args.push('--disable-dev-shm-usage');
      */
    }
    return launchOptions;
  });

First, these are chrome options that get passed to Chrome when Cypress starts it, so they only apply if the browser you’re using for the test is Chrome. If you do a search for chrome options you can pass in, you’ll see that proxy-bypass-list is telling Chome “hey, that proxy I told you about, don’t send things for these hosts/hosts:ports through it.” But you notice I didn’t set a proxy, and the Chrome docs say this option only takes effect if you set a proxy-server. So where is the proxy coming from? Cypress. If you console.log(JSON.stringify(launchOptions)), you’ll see there is a localhost:someweirdport proxy being passed in and if you open your browser and go to it, it’s Cypress. So I think this is how Cypress is intercepting everything so you can mock it, etc. By passing in this “proxy-bypass-list” we’re preventing Cypress from intercepting certain things, which is what we want to do for Firestore (localhost:8080 is the port we have our Firestore emulator running on).

Since by default running cypress in headed mode runs in Chrome, passing the localhost:8080 to chrome in the proxy-bypass-list was enough to allow me to comment out experimentalForceLongPolling:true, and the tests in headed mode worked immediately. Which was awesome because I could never get them to even partially work before without experimentalForceLongPolling:true.

By default Cypress runs Electron in headless mode though, so our automated tests would still fail without experimentalForceLongPolling:true, just as they always had. So I passed the --browser chrome option into the cypress run command in our package.json to tell Cypress to run against Chrome in headless mode too, and voila, it worked there too!

But now there was a big problem when running in our CI/CD pipeline. We use cucumber, and any cucumber feature file with more than a couple of tests would fail. It would work fine on my machine in headless mode, but fail in the CI/CD pipeline (we’re limited to 8 gigs in our CI/CD pipeline VM’s). I added the DEBUG=cypress:* flag to be before the cypress run command in package.json (we’re using cross-env, not sure if this works without that or if you’d have to set this variable manually in a terminal). Adding that Debug flag prints out the memory being used by Cypress periodically in a nice chart, and breaks it down between the browser (Chrome in this case) and other things being spun up by Cypress. And boy is Chrome a memory hog, which is what was causing any Cucumber feature file with more than a few tests to fail by causing out of memory errors in our CI/CD VM’s.

So then I thought maybe I could go back to Electron and pass in a similar proxy-bypass-filter option to Electron. But although there is some documentation saying you can do this with an environment variable (here and here), I couldn’t get it to work. I tried setting it in package.json as part of the cross-env command and also as a straight up terminal variable before I ran the package.json command, and it just didn’t seem to take effect. It wasn’t the only thing: for that DEBUG=cypress:* flag I mentioned above you can pass in options to debug specific things (like memory), but that never seemed to take effect either - the only way it would work is with DEBUG=cypress:*, which prints out every debug message under the sun.

So for now we are left splitting our feature files that are causing out of memory issues on our CI/CD pipeline VM’s into multiple feature files. Apparently after every feature file cypress-cucumber resets the browser, clearing the memory. Sometimes we’re having to split a file with 2 tests into 2 files with 1 test each. But that seems a small price to pay to have the experimentalForceLongPolling option disabled. Our tests are reliably passing, and run so much faster now we’re finding things that were flaky we’d never noticed before because our tests ran so ridiculously slow.

I’m hoping this is just a first step. I’d like to put that DEBUG option back on with Electron and the experimentalForceLongPolling turned back on to see how much memory Electron is consuming compared to Chrome, because we never had these memory issues with Electron. And I’m hoping now that I’ve gotten this far, Cypress may be able to provide some guidance about how to get passing the proxy bypass into Electron to work.

This issue has been under development on and off for a year now. Is there any update on deliverability? My team is testing two apps that are dependent on Firebase and we would really like for Cypress to be the tool we use for these.

I work on Firestore, so I can confirm: the default behavior of Firestore’s web SDK is to make use of WebChannel’s streaming mode. The client makes what looks like an XHR, but then the server will hold the response open for 60 seconds and send as many server-initiated responses as it can during that time window.

The experimentalForLongPolling option forces the server to send only a single response per request. While this works around issues where proxies buffer responses, it also introduces a 1/2 RTT delay between every response. You won’t notice this developing your app in the US talking to a Firestore instance in us-central, but RTT from many parts of the world to the same instance is over 250 ms. Don’t just blanket enable this option if you can avoid it.

I have this issue as well.

Doing

  if (window.Cypress) {
        Firebase.firestore().settings({ experimentalForceLongPolling: true })
    }

solved it

Is there an appropriate solution to this issue?

We’re still able to replicate the issue with the repo that was originally provided and with the Firestore quickstart project, but still need some time to figure out a fix.

It’s in a queue with a couple of other high priority, but pretty time consuming issues that we’ll get to as soon as possible.

For those looking for a repro of not being able to see Firestore emulator in Cypress, which seems to be similar/related, I provide one in this issue: https://github.com/cypress-io/cypress/issues/6350.

In an issue opened in firebase-tools, it was mentioned that passing experimentalForceLongPolling: true to firestore settings that seems help. When setting this in the above mentioned repro, the data loads!

There was also the following theory from @wvanderdeijl that seems to make sense:

I think the underlying problem is that Cypress is intercepting all network traffic so it can monitor and sometimes mock. However, the webchannel protocol used by firestore has multiple replies over the same http request. The Cypress code cannot handle this and will only forward the first reply and ignore the rest.

^ Also aligns with what @mslooten mentioned above (so it seems that it isn’t specific to AngularFire, but instead just Firestore in general)

I think we might be having the same issue. Unfortunately also private source code, but perhaps some observations could be of use?

• Angular
• Firestore
• Cypress

When running the tests, I can see the Firestore requests showing up in the sidebar as XHR. However, they never respond when running in Cypress. When I open the url in my browser, I can see the network calls respond. The app is still working in Cypress, although the requests will show as pending and eventually timeout. It also means I cannot use .wait(), because they never return a value. I suppose I could try to stub them, but of course we rather have a choice.

I’m starting to think it has something to do with the custom protocol in use. It seems Firestore uses something called WebChannels, or at least AngularFire does (https://google.github.io/closure-library/api/goog.net.WebChannel.html).

I could be way wrong here, but I guess it makes sense that IF a custom protocol is used, Cypress won’t be able to intercept it the way it does XHR reqs (similar to .fetch perhaps?). Please let me know if there’s anything I can do to help move this forward.

Thank you for looking at this and for this awesome product. I really want to use it but our web app is heavily using Firestore. Any idea when this could be fixed? I need to decide soon which testing framework should we use… 😦

Thanks again =)

Any plans for when to address the mentioned performance issues?

@mslooten if the Network tab says it’s going out as an XHR, we should be able to intercept it

Does your company require a proxy to access the Internet, by chance? If so, that’s probably what’s causing the error. Currently HTTPS requests to third-party domains behind a proxy will time out.

Cypress 3.3.0 will contain fixes for #1469 that will allow this proxied traffic to reach the web and work correctly.

First, thanks for the quick reply. Appreciate it! This behavior is consistent whether on company or home network (no proxies). However, the application is served from localhost (http://localhost:4200) and Firestore is HTTPS, not sure if that’s the same thing though.

I can see the XHR requests being made, I can even see the application updating with the right data (that was obviously returned from Firestore), but the requests stay pending in Cypress. Visiting the same url in my browser (also Chrome), I can see the XHR requests and the responses.

I think PR #1195 addresses this, it just needs a review

FYI, with firebase version 9 you have to use initialFirestore to opt-in the settings, like so const firestore = initializeFirestore(app, { experimentalForceLongPolling: true }) And this must be called before any further access to firestore, including connectFirestoreEmulator(firestore, host, port)

To only enabled in localhost, maybe you can do this

const db = firebase.firestore();

if (location.hostname === "localhost") {
  db.settings({
    experimentalForceLongPolling: true,
  });
}

Thank you @mesqueeb for an example setup. Thank you @wilhub for confirming it; confirmation from someone close to the code helps. Is there any thoughts from cypress team about working with/around this? Firestore may remove this flag as it’s experimental; besides would it be possible for Cypress to handle it auto-magically without user googling around for the problem.

Any update on this? My company’s project is entirely serviced by firestore, and we were really hoping to use cypress to do our integration tests.

Andres, do you have full reproduction example that shows the problem?

Sent from my iPhone

On Nov 25, 2018, at 21:13, Andres Castano notifications@github.com wrote:

Any news on this? I am afraid I can’t wait any longer… 😢

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

Any news on this? I am afraid I can’t wait any longer… 😢

It’s on npm: https://www.npmjs.com/package/cypress

I can confirm this still doesn’t work in the latest develop branch. If anyone requires help reproducing let me know, but I think simple running any kind of firestore query will do the trick - the query never completes. Hopefully this will be get looked at soon, as we our company will soon be forced to choose another testing solution 😦

@bentlusty please confirm the version used -develop branch ? 3.1.1 ?

The PR didn’t fix it, all firestore requests time out