cypress: No longer fail runs based on recording limit

After reading feedback provided from this thread and discussing requirements needed to implement this change with our team, we have decided to change how we handle recording limits being reached for free tier plans - to no longer fail and exit the run. The new behavior will reflect how the recording limit is handled for paid plans, the run will exit normally and viewing data will be limited within the Dashboard views.

We don’t have a clear estimate on when this will be delivered, but we wanted to update everyone to know that there is work planned and to reopen the issue.

Thank you all for providing feedback. We welcome criticism on the product decisions we make and want you all to be successful in testing with Cypress.

@tennisgent Tests that exceed the test recordings limit on the free plan do not record. If you are within the limit at the beginning of the cypress run command, the recordings should continue as normal, even if you exceed the test limit, but upon exceeding it, the next cypress run command will no longer run the tests in CI and instead show this error.

This only happens on the free plan. The test recordings are always recorded on any of our paid plans or our OSS plan, which offers unlimited test recordings for open source projects.

To continue running your test suite, you can upgrade to a paid plan or apply for an OSS plan (if eligible) or remove the --record flag from cypress run.

This is intended behavior that we don’t plan to change at the moment.

Thank you for the feedback. I will forward this along to our team. We’re always looking for ways to improve our pricing offerings and your feedback is a big part of that evaluation.

I find this to be a total dark pattern - it seems like the only reason this is done is to convince people to convert to paid customers to unbreak their build. A warning is fine, even a loud and pronounced one, but to completely exit tests if you run out of recordings is absurd. I should not have to commit changes to my CI config because I ran out of recordings.

I’m pretty disappointed, since I use Cypress at work and was excited to use it on a side project. I enabled dashboard recordings since I usually stay under 500 tests a month, and figured there wouldn’t be any harm in hitting the limit anyways - and now I see this is not the case 😦

Since Cypress doesn’t have any plan between 0 and 100 dollars a month, and my side project is (a) noncommercial and (b) not under an OSI-approved license, there’s not really any path forward here other than either removing Cypress recordings or using one of these wrapper scripts. I get not wanting to give stuff away for free, and wouldn’t have even minded if y’all just didn’t have a free plan, or only had a timed trial, but I think breaking builds when the free limits are hit is the worst imaginable experience I could have with Cypress.

Recorded runs that reach the limit within a free tier will no longer exit the run. When the test recording limit is reached - a warning message will display within the stdout like shown below, then continue on to running and recording the run. Admins and owners of the organization will also get an email warning them of the overage.

We will probably tweak this warning message to be a little nicer in a future versioned release of the Test Runner.

Upon going to view the data within the Dashboard, the run data will be hidden and parallelization will be disabled until the organization upgrades to a paid plan. This is the exact same behavior as any other paid plan that reaches their test recording limit.

Thanks everyone for your feedback! If you encounter any issues with recording in the free tier, please make sure to open a new issue detailing the problem.

Closing this issue as resolved.

Hi @jennifer-shehane. Thanks for your reply. The point I’m trying to make is that stopping CI runs because the recording limit is too disruptive.

Notifying folks via email / dashboard messages is great! I did see the message on the dashboard, but I did not received an email, btw.

My suggestion is to keep alerting via email and the dashboard, but to still run the tests, albeit without recording it.

Thank you!

I was also quite surprised to realize this is how Cypress handles reaching the free-quota. I would also strongly suggest to change this. Just show a note (“Plan limit reached, these tests will no be recorded”) and run the tests nonetheless.

Also I do not think that this will influence the developers experience in any way. You get an email, you get a banner on the dashboard and a message in the CI. If that is not enough to clarify the situation, then I think someone chose the wrong job 😉

@dotherightthing I ended up coming up with this script. It does the check Brian suggested, passes params and returns the correct error codes:

https://gist.github.com/edelgado/9dbceac4e275c0579fe91170ecc35645

I find the language here confusing and agree with the others that the tests should run irrespective of whether the video is stored / sent to https://dashboard.cypress.io. Running tests and recording them are two separate things. I like the videos but I can live without them, but Cypress has zero value if I can’t see my test results in the Terminal.

@tennisgent what you’re asking for can actually be achieved today using the module API programmatically: https://on.cypress.io/module-api

Here’s some sample code…

const cypress = require('cypress')

const didRunError = (results) => {
  return results.message && results.message.includes('Could not find Cypress test run results')
}

// run using our module API
cypress.run({ record: true })
.then((results = {}) => {
  if (didRunError(results)) {
    return cypress.run({ record: false })
  }
})

The code in the module API that specifically handles this is here: https://github.com/cypress-io/cypress/blob/master/cli%2Flib%2Fcypress.js#L31

Under the hood - if the run errors out before running tests it will not generate any test results, and therefore we resolve with the object as described above.

However, I’ve opened a new issue to make this a little easier - for instance we should probably capture the underlying error reason why Cypress abandoned the run, and then we should probably reject the promise under the hood rather than resolve it with an arbitrary object.

Let me join your friendly talk 😃

From what I’ve read above, it does not seem like you listen to your customers (which actually works against you, because the worst customer — is an annoyed customer). Instead of trying to find a common solution, you just ignore everyone here.

If you don’t want to change the default behavior — it’s fine, but at least add an option to override this behavior. I don’t want to remove --record and commit this change every time just because of your bad UX. I like your product, but this solution can be improved in favor of your customers.

You could add an option like this:

cypress run --record --key ... --mute-record-failures

And everyone here will be happy 😃