coolify: [Bug]: /artifacts/build.sh times out attempting to load .dockerignore & Dockerfile

Description

Using Coolify Cloud & v4.0.0-beta.323

All deployments are suddenly stuck loading at & time out after a long while:

[2024-Aug-20 10:05:11.823831] ----------------------------------------
[2024-Aug-20 10:05:11.833426] Building docker image started.
[2024-Aug-20 10:05:11.839621] To check the current progress, click on Show Debug Logs.
[2024-Aug-20 10:05:17.033470]

[COMMAND] docker exec jw4k848wo8s0og0k8kgo08s8 bash -c 'bash /artifacts/build.sh'
[OUTPUT] #0 building with "default" instance using docker driver  #1 [internal] load .dockerignore  
[2024-Aug-20 10:05:26.929672]

[COMMAND] docker exec jw4k848wo8s0og0k8kgo08s8 bash -c 'bash /artifacts/build.sh'
[OUTPUT] #1 ...  #2 [internal] load build definition from Dockerfile

Revalidating server works fine but doesn’t fix the issue. I’ve raised this issue in the Discord and one more user has the same sudden issue so far.

Minimal Reproduction (if possible, example repository)

Deploying any Dockerfile on Coolify Cloud (tried Nixpacks aswell)

Exception or Error

No response

Version

v4.0.0-beta.323

Cloud?

  • Yes
  • No

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 36 (8 by maintainers)

Most upvoted comments

Same problem

It is temporarily solved by restarting docker but after a while it starts to give problems again

Hi. I’m on the last beta (335) and 2 apps are stuck in “load build definition from Dockerfile”. I have enough disk space. This is a production server so I can’t restart docker. @andrasbacsai Any suggestion ? Thanks.

As I checked, the builder cache of Docker builds up to a certain point, even Docker cannot handle it. So you meed to restart Docker Engine, and then cleanup (docker builder prune) manually.

Probably Nixpacks fills up the cache this much. ☹️

I also hit this issue last night. The server deployed in the morning and failed to deploy later in the day.

Tried to run the prune command, docker builder prune -af but it would not complete.

Forced the Docker service to restart, sudo systemctl restart docker, and it was able to. Took about 2-3 minutes. I then ran the prune command and it completed in 5 seconds or so.

The server is not critical but it will be a nuisance if this is a going concern. It would be really nice if there was some number to watch to know when this is going to happen. Ideally that number is tied to an automated fix or clean up that prevents me from seeing the problem.

Problem started on beta 370. I upgraded to beta 373 and the problem persisted.

Yes it is fixed an cache will be cleaned by the coolify job if force cleanup is enabled or soon via a button click to run it manually. The problem is if build cache execeds a certain amount docker can not handel it anymore so the only fix is to restart docker. See the comment here https://github.com/coollabsio/coolify/issues/3150#issuecomment-2326573453

The problem is that this command also hangs though. Running this manually did not work in the cases that I and others were reporting. Leaving a restart or reinstall as the only option left.

I think the issue is that the automatically running cleanup job from coolify was not working properly - leading to the huge cache - leading to no commands working.

Your disk is probably full. Can you please check?

The next version of Coolify will have a fix for the automated docker cleanup job.