ddev: Large project fails to start (mutagen sync times out?) (huge node_modules)
Is there an existing issue for this?
- I have searched the existing issues
Output of ddev debug test
Expand `ddev debug test` diagnostic information
Running bash [-c /var/folders/0m/tg6xfqy14mn4lnws2jmstr4w0000gr/T/test_ddev.sh]
======= Existing project config =========
These config files were loaded for project myproject: [/Users/dmckenna/Sites/myproject/.ddev/config.yaml]
name: myproject
type: drupal9
docroot: web
php_version: 8.0
webserver_type: nginx-fpm
webimage: ddev/ddev-webserver:20230803_php_serialize_precision
router_http_port: 80
router_https_port: 443
additional_hostnames: []
additional_fqdns: [local.pe.gatech.edu]
mariadb_version: 10.3
database: {mariadb 10.3}
hooks: map[post-import-db:[map[exec:drush cr] map[exec:drush en -y stage_file_proxy] map[exec:drush sql-sanitize -y] map[exec:drush uli]] post-pull:[map[exec:drush cr] map[exec:drush en -y stage_file_proxy] map[exec:drush sql-sanitize -y] map[exec:drush uli]] post-start:[map[exec:mysql -uroot -proot -e "CREATE DATABASE IF NOT EXISTS d7; GRANT ALL ON d7.* to 'db'@'%';" service:db]]]
mailhog_port: 8025
mailhog_https_port: 8026
project_tld: ddev.site
use_dns_when_possible: true
composer_version: 2
nodejs_version: 18
default_container_timeout: 120
======= Creating dummy project named tryddevproject-25584 in ../tryddevproject-25584 =========
OS Information: Darwin DMcKennaWork.local 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000 arm64
ProductName: macOS
ProductVersion: 13.5.1
BuildVersion: 22G90
User information: uid=504(dmckenna) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),703(com.apple.sharepoint.group.3),702(com.apple.sharepoint.group.2),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh),400(com.apple.access_remote_ae),701(com.apple.sharepoint.group.1)
DDEV version: ITEM VALUE
DDEV version v1.22.1
architecture arm64
db ddev/ddev-dbserver-mariadb-10.4:v1.22.0
ddev-ssh-agent ddev/ddev-ssh-agent:v1.22.0
docker 24.0.5
docker-compose v2.20.0
docker-platform orbstack
mutagen 0.17.2
os darwin
router ddev/ddev-router:v1.22.0
web ddev/ddev-webserver:20230803_php_serialize_precision
PROXY settings: HTTP_PROXY='' HTTPS_PROXY='' http_proxy='' NO_PROXY=''
======= DDEV global info =========
Global configuration:
instrumentation-opt-in=false
omit-containers=[]
performance-mode=mutagen
router-bind-all-interfaces=false
internet-detection-timeout-ms=3000
disable-http2=false
use-letsencrypt=false
letsencrypt-email=
table-style=default
simple-formatting=false
auto-restart-containers=false
use-hardened-images=false
fail-on-hook-fail=false
required-docker-compose-version=v2.20.0
use-docker-compose-from-path=false
project-tld=
xdebug-ide-location=
no-bind-mounts=false
router=nginx-proxy
wsl2-no-windows-hosts-mgt=false
router-http-port=80
router-https-port=443
traefik-monitor-port=10999
======= DOCKER info =========
docker location: lrwxr-xr-x 1 dmckenna admin 34 Jul 31 08:16 /opt/homebrew/bin/docker -> ../Cellar/docker/24.0.5/bin/docker
Docker Desktop Version: Print: Entry, ":CFBundleShortVersionString", Does Not Exist
Print: Entry, ":CFBundleVersion", Does Not Exist
Docker Desktop for Mac File build Doesn'tDocker Desktop for Mac Exist, build WillDocker Desktop for Mac Create: build /Applications/Docker.app/Contents/Info.plistDocker Desktop for Mac File build Doesn'tDocker Desktop for Mac Exist, build WillDocker Desktop for Mac Create: build /Applications/Docker.app/Contents/Info.plist
docker version:
Client: Docker Engine - Community
Version: 24.0.5
API version: 1.43
Go version: go1.20.6
Git commit: ced0996600
Built: Wed Jul 19 19:44:22 2023
OS/Arch: darwin/arm64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.5
API version: 1.43 (minimum version 1.12)
Go version: go1.20.6
Git commit: a61e2b4
Built: Fri Jul 21 20:35:24 2023
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: v1.7.3
GitCommit: 0cae528dd6cb557f7201036e9f43420650207b58
runc:
Version: 1.1.9
GitCommit: 82f18fe0e44a59034f3e1f45e475fa5636e539aa
docker-init:
Version: 0.19.0
GitCommit: de40ad0
DOCKER_DEFAULT_PLATFORM=notset
======= Mutagen Info =========
Mutagen is installed in ddev, version=0.17.2
--------------------------------------------------------------------------------
Name: alv-alvernia
Identifier: sync_GZZR2xFrI5tkTfeakG18rzbrYVlAjsgIOjubIOFUKtY
Labels:
com.ddev.config-hash: c212281329fb24762a96e1b7a37a19ed84868866
com.ddev.volume-signature: Users-dmckenna-orbstack-run-1692987843
Configuration:
Synchronization mode: Two Way Resolved
Hashing algorithm: Default (SHA-1)
Maximum allowed entry count: Default (2⁶⁴−1)
Maximum staging file size: Default (18 EB)
Symbolic link mode: POSIX Raw
Ignore VCS mode: Default (Propagate)
Ignores:
/.git
/.tarballs
/.ddev/db_snapshots
/.ddev/.importdb*
.DS_Store
.idea
/web/sites/default/files
Permissions mode: Default (Portable)
Alpha:
URL: /Users/dmckenna/Sites/alv_alvernia
Configuration:
Watch mode: Default (Portable)
Watch polling interval: Default (10 seconds)
Probe mode: Default (Probe)
Scan mode: Default (Accelerated)
Stage mode: Neighboring
File mode: Default (0600)
Directory mode: Default (0700)
Default file/directory owner: Default
Default file/directory group: Default
Connected: Yes
Synchronizable contents:
15396 directories
65262 files (522 MB)
85 symbolic links
Beta:
URL: docker://ddev-alv-alvernia-web/var/www/html
DOCKER_HOST=unix:///Users/dmckenna/.orbstack/run/docker.sock
Configuration:
Watch mode: Default (Portable)
Watch polling interval: Default (10 seconds)
Probe mode: Default (Probe)
Scan mode: Default (Accelerated)
Stage mode: Neighboring
File mode: Default (0600)
Directory mode: Default (0700)
Default file/directory owner: Default
Default file/directory group: Default
Compression: Default (DEFLATE)
Connected: Yes
Synchronizable contents:
15396 directories
65262 files (522 MB)
85 symbolic links
Status: Watching for changes
--------------------------------------------------------------------------------
Name: myproject
Identifier: sync_SyoiEDUt8aG0Lpk1fqV2KioTHKMpSBfk1tYKvU6MkhG
Labels:
com.ddev.config-hash: da39a3ee5e6b4b0d3255bfef95601890afd80709
com.ddev.volume-signature: Users-dmckenna-orbstack-run-1692990778
Configuration:
Synchronization mode: Default (Two Way Safe)
Hashing algorithm: Default (SHA-1)
Maximum allowed entry count: Default (2⁶⁴−1)
Maximum staging file size: Default (18 EB)
Symbolic link mode: Default (Portable)
Ignore VCS mode: Default (Propagate)
Ignores: None
Permissions mode: Default (Portable)
Alpha:
URL: /Users/dmckenna/Sites/myproject
Configuration:
Watch mode: Default (Portable)
Watch polling interval: Default (10 seconds)
Probe mode: Default (Probe)
Scan mode: Default (Accelerated)
Stage mode: Default (Mutagen Data Directory)
File mode: Default (0600)
Directory mode: Default (0700)
Default file/directory owner: Default
Default file/directory group: Default
Connected: No
Beta:
URL: docker://ddev-myproject-web/var/www/html
DOCKER_HOST=unix:///Users/dmckenna/.orbstack/run/docker.sock
Configuration:
Watch mode: Default (Portable)
Watch polling interval: Default (10 seconds)
Probe mode: Default (Probe)
Scan mode: Default (Accelerated)
Stage mode: Default (Mutagen Data Directory)
File mode: Default (0600)
Directory mode: Default (0700)
Default file/directory owner: Default
Default file/directory group: Default
Compression: Default (DEFLATE)
Connected: No
Status: [Paused]
--------------------------------------------------------------------------------
======= Docker Info =========
Docker platform: orbstack
Using docker context: default (unix:///Users/dmckenna/.orbstack/run/docker.sock)
docker-compose: v2.20.0
Using DOCKER_HOST=unix:///Users/dmckenna/.orbstack/run/docker.sock
Docker version: 24.0.5
Able to run simple container that mounts a volume.
Able to use internet inside container.
Docker disk space:
Filesystem Size Used Available Use% Mounted on
overlay 414.6G 6.9G 407.8G 2% /
PhpMyAdmin (`dba`) is no longer part of DDEV core, please edit your `omit_containers` configuration to remove it
Container ddev-alv-alvernia-db Stopped
Container ddev-alv-alvernia-web Stopped
Container ddev-alv-alvernia-redis Stopped
Container ddev-alv-alvernia-solr Stopped
Container ddev-alv-alvernia-web Stopped
Container ddev-alv-alvernia-db Stopped
Container ddev-alv-alvernia-db Removed
Container ddev-alv-alvernia-web Removed
Container ddev-alv-alvernia-solr Stopped
Container ddev-alv-alvernia-redis Stopped
Container ddev-alv-alvernia-redis Removed
Container ddev-alv-alvernia-solr Removed
Network ddev-alv-alvernia_default Removed
Project alv-alvernia has been stopped.
Stopped mutagen daemon
The ddev-ssh-agent container has been removed. When you start it again you will have to use 'ddev auth ssh' to provide key authentication again.
Existing docker containers:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Network ddev_default created
Starting tryddevproject-25584...
Container ddev-ssh-agent Created
Container ddev-ssh-agent Started
ssh-agent container is running: If you want to add authentication to the ssh-agent container, run 'ddev auth ssh' to enable your keys.
Network ddev-tryddevproject-25584_default Created
Container ddev-tryddevproject-25584-web Created
Container ddev-tryddevproject-25584-db Created
Container ddev-tryddevproject-25584-web Started
Container ddev-tryddevproject-25584-db Started
You have Mutagen enabled and your 'php' project type doesn't have `upload_dirs` set.
For faster startup and less disk usage, set upload_dirs to where your user-generated files are stored.
If this is intended you can disable this warning with `ddev config --disable-upload-dirs-warning`.
Starting mutagen sync process... This can take some time.
Mutagen sync flush completed in 1s.
For details on sync status 'ddev mutagen st tryddevproject-25584 -l'
Container ddev-router Created
Container ddev-router Started
Successfully started tryddevproject-25584
Project can be reached at https://tryddevproject-25584.ddev.site https://127.0.0.1:32782
======== Curl of site from inside container:
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 25 Aug 2023 19:48:11 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
======== curl -I of http://tryddevproject-25584.ddev.site from outside:
HTTP/1.1 200 OK
Server: nginx/1.20.1
Date: Fri, 25 Aug 2023 19:48:11 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
======== full curl of http://tryddevproject-25584.ddev.site from outside:
Success accessing database... db via TCP/IP
ddev is working. You will want to delete this project with 'ddev delete -Oy tryddevproject-25584'
======== Project ownership on host:
drwxr-xr-x 4 dmckenna staff 128 Aug 25 15:47 ../tryddevproject-25584
======== Project ownership in container:
drwxr-xr-x 1 dmckenna dialout 16 Aug 25 19:48 /var/www/html
======== In-container filesystem:
Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/vdb1 btrfs 433738468 7339072 426408740 2% /var/www
======== curl again of tryddevproject-25584 from host:
Success accessing database... db via TCP/IP
ddev is working. You will want to delete this project with 'ddev delete -Oy tryddevproject-25584'
Thanks for running the diagnostic. It was successful.
Please provide the output of this script in a new gist at gist.github.com
Running ddev launch in 5 seconds
Please run cleanup after debugging with 'ddev debug testcleanup'
Expected Behavior
ddev start should work.
Actual Behavior
Running ddev start on a project with ~70,000 files (per find . -type f | wc -l) works fine.
Running ddev start on a project with ~150,000 files fails.
Steps To Reproduce
macOS Ventura 13.5.1. MacBook Pro M1/Max. Colima installed via Homebrew. ddev configured to use Mutagen (default). All software running the latest. Colima configured to use vz and all vz-related options. Start a project with a large codebase and Drupal sites/default/files directory.
Anything else?
$ ddev start
Starting myproject...
User-managed /Users/dmckenna/Sites/myproject/.ddev/.gitignore will not be managed/overwritten by ddev
Using custom nginx configuration in /Users/dmckenna/Sites/myproject/.ddev/nginx_full/nginx-site.conf
Using nginx snippets: [/Users/dmckenna/Sites/myproject/.ddev/nginx/file-redirect.conf /Users/dmckenna/Sites/myproject/.ddev/nginx/patternlab.conf]
Using custom mysql configuration: [/Users/dmckenna/Sites/myproject/.ddev/mysql/migration.cnf]
Custom configuration is updated on restart.
If you don't see your custom configuration taking effect, run 'ddev restart'.
Network ddev-myproject_default Created
Container ddev-myproject-solr Created
Container ddev-myproject-db Created
Container ddev-myproject-redis Created
Container ddev-myproject-web Created
Container ddev-myproject-redis Started
Container ddev-myproject-solr Started
Container ddev-myproject-db Started
Container ddev-myproject-web Started
Starting mutagen sync process... This can take some time.
..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Mutagen sync completed with problems in 24m39s.
For details on sync status 'ddev mutagen st myproject -l'
Failed waiting for web/db containers to become ready: web container failed: log=[{2023-08-25 15:37:35.843279987 -0400 EDT 2023-08-25 15:37:35.889710137 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED } {2023-08-25 15:37:36.901121274 -0400 EDT 2023-08-25 15:37:36.948376739 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED } {2023-08-25 15:37:37.961590591 -0400 EDT 2023-08-25 15:37:38.00559451 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED } {2023-08-25 15:37:39.017562329 -0400 EDT 2023-08-25 15:37:39.062703237 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED } {2023-08-25 15:37:40.072686586 -0400 EDT 2023-08-25 15:37:40.105707756 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED }], err=container /ddev-myproject-web unhealthy: [{2023-08-25 15:37:35.843279987 -0400 EDT 2023-08-25 15:37:35.889710137 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED } {2023-08-25 15:37:36.901121274 -0400 EDT 2023-08-25 15:37:36.948376739 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED } {2023-08-25 15:37:37.961590591 -0400 EDT 2023-08-25 15:37:38.00559451 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED } {2023-08-25 15:37:39.017562329 -0400 EDT 2023-08-25 15:37:39.062703237 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED } {2023-08-25 15:37:40.072686586 -0400 EDT 2023-08-25 15:37:40.105707756 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED }]
Container ddev-router Running
Failed to start myproject: container(s) failed to become healthy before their configured timeout or in 120 seconds. This may be just a problem with the healthcheck and not a functional problem. (container /ddev-myproject-web is unhealthy: [{2023-08-25 15:37:37.961590591 -0400 EDT 2023-08-25 15:37:38.00559451 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED } {2023-08-25 15:37:39.017562329 -0400 EDT 2023-08-25 15:37:39.062703237 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED } {2023-08-25 15:37:40.072686586 -0400 EDT 2023-08-25 15:37:40.105707756 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED } {2023-08-25 15:37:41.115423556 -0400 EDT 2023-08-25 15:37:41.147059105 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED } {2023-08-25 15:37:42.157884935 -0400 EDT 2023-08-25 15:37:42.222359774 -0400 EDT 1 /var/www/html:OK mailhog:FAILED phpstatus:FAILED }])
About this issue
- Original URL
- State: closed
- Created 10 months ago
- Comments: 29 (29 by maintainers)
No, I think you probably had a custom nginx-site.conf, since DDEV detected one and warned you about it, and that could definitely cause this problem.
Please make sure you don’t have the altered .ddev/nginx-full/nginx-site.conf checked in as it will just reappear of course.
I’m glad you have sorted it out! I’m going to take a look at your PR to docs to see if we can make it more general. Sometimes people have large fonts directories, or database dump directories, etc. Some of those need to be inside the container, others don’t.
It turned out the Mutagen problem stemmed from the node_modules directory used with the theme; I added “/web/themes/custom/MYTHEME/node_modules” to the sync:defaults:ignore:paths part of the mutagen.yml file and it became super fast:
What a dramatic difference!
I’m going to recommend we set this up at work on all of our projects, it’ll save an awful lot of pain!