coolify: [Bug]: Upgrading coolify version causes scheduled database backups to fail

Description

After upgrading the coolify version, scheduled database backup start failing. It looks like it’s unable to pull the coolify-helper image from ghcr and retries to do so at an impressive polling rate of 271.469µs, quickly hitting the github API rate limit?

This is fixed by triggering a manual backup, after which scheduled backups start working as expected. Note that I have only one database running on my VPS, I’m not sure how it would behave with more than one. I’ve recently started using Coolify and I’ve upgraded twice so far, this issue appeared both times.

Minimal Reproduction (if possible, example repository)

  1. Create a new project and a stage environment for it.
  2. Create a MongoDB database in stage with public access enabled.
  3. Configure S3 storage using R2 Cloudflare
  4. Setup automatic backups to trigger every hour and upload to S3
  5. Upgrade the Coolify version using the UI
  6. Database backups start failing
  7. Trigger a manual backup to fix this issue.

Exception or Error

Unable to find image ‘ghcr.io/coollabsio/coolify-helper:latest’ locally docker: Error response from daemon: toomanyrequests: retry-after: 271.469µs, allowed: 44000/minute. See ‘docker run --help’.

DatabaseBackupJob failed with: Unable to find image ‘ghcr.io/coollabsio/coolify-helper:latest’ locally latest: Pulling from coollabsio/coolify-helper 3c854c8cbf46: Pulling fs layer d2aac0aefc71: Pulling fs layer f9c396f433c8: Pulling fs layer a9b023bc0a40: Pulling fs layer 74b8f2dbaf15: Pulling fs layer 4f4fb700ef54: Pulling fs layer 616e93ebdc58: Pulling fs layer 4f4fb700ef54: Waiting 616e93ebdc58: Waiting a9b023bc0a40: Waiting docker: error pulling image configuration: download failed after attempts=1: toomanyrequests: retry-after: 1.296341ms, allowed: 44000/minute. See ‘docker run --help’.

Version

329

Cloud?

  • Yes
  • No

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

I have made a few improvements in the upcoming version. The helper image won’t be pruned by Coolify’s cleanup process, so it will be downloaded once.

Also the default cleanup schedule will be changed to run at midnight, not every 10 minutes (which is a bit agressive).

Edit: Will be released on Monday morning.

Released a new version with this fix. Can you please update and let us know if it still occurs?

Thanks @peaklabs-dev for handling this on the weekend!

@peaklabs-dev I can confirm that this was fixed with 332 for my instance 🎉

After updating, it failed again, this time by logging the following error twice

Failed to pull helper image: Error response from daemon: toomanyrequests: retry-after: 367.879µs, allowed: 44000/minute

Failed to pull helper image: Error response from daemon: toomanyrequests: retry-after: 367.879µs, allowed: 44000/minute

I “fixed” it once again by triggering a manual backup, after which scheduled backups work again. I’ll post an update next time I upgrade coolify’s version or if this occurs again in another context.

This is fixed. Just waiting for andras to release the new version. https://github.com/coollabsio/coolify/pull/3353

I also noticed that the same issue occurs when the Force Docker Cleanup option is turned on for the server that the database is on, which I guess is kind of expected, but I’m adding it here for the sake of completeness.

Just found out that a simmilar rate limiting issue was already reported here: https://github.com/coollabsio/coolify/issues/2930