compose: [BUG] `watch` `sync` does not pass project files to container / does not reflect file changes between container restarts

Description

I’ve wanted to try the newest docker compose watch (released with latest desktop), but I have two issue with it. The announcement blog post seems to suggest, that adding a config such as this one in an example repository and running docker compose watch should result in a container named app and the folder /var/www/html should have the contents of the current directory.

Thing is, the directory is empty and only new changes to files are copied over.

This lead me to believe, that I would need to add COPY . /var/www/html to Dockerfile so files are copied in on image build, but this creates a second problem, as any changes I made to project files since image build are not reflected to the container between container restarts (say across multiple days of development).

Anyway, here is a repository with an example setup, including steps to reproduce https://github.com/edvordo/docker-compose-watch-test

I’m more than willing to concede I’m reading the docs wrong, but I’ve tried support channels like the (unofficial?) discord server and we could not figure out what could be wrong.

The issue is the same using the official examples provided here: https://github.com/dockersamples/avatars

While yes, the containers will have content after first build (thanks to the COPY instruction), they will not have the changes made to the project files between container restarts:

while watch is running, change <title> in web/index.html to whatever
changes are reflected on localhost:5735 and in container
kill watch, run docker compose down and run docker compose watch again
<title> in browser / container will NOT have your changes

Steps To Reproduce

No response

Compose Version

compose version
Docker Compose version v2.22.0-desktop.2

docker-compose version
Docker Compose version v2.22.0-desktop.2

Docker Environment

Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     /Users/edvordo/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.22.0-desktop.2
    Path:     /Users/edvordo/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/edvordo/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/edvordo/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.8
    Path:     /Users/edvordo/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/edvordo/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/edvordo/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.7
    Path:     /Users/edvordo/.docker/cli-plugins/docker-scout

Server:
 Containers: 10
  Running: 6
  Paused: 0
  Stopped: 4
 Images: 59
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 3.838GiB
 Name: docker-desktop
 ID: 73a03b6a-4caf-49e5-947b-af0d67cde38e
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

About this issue

Original URL
State: open
Created 8 months ago
Reactions: 5
Comments: 17 (2 by maintainers)

Most upvoted comments

I don’t have a problem with watch per-se, more of the action: sync part of the service configuration.

It seems highly impractical to rebuild the container(s) every single time I bring up the containers just so that I get to have changes in since last build. And I can’t volume in the directory, because then action: sync / watch will complain, that the directory is already in and won’t work.

WARN[0002] path '/path/to/project/docker-compose-watch-test' also declared by a bind mount volume, this path won't be monitored!

I’m looking for a middle ground really, have sync pull / copy the directory into the container on up, like a volume / mount would and have watch still sync in files as I change them.

edvordo on Oct 18, 2023

comparing last updated date require a docker API access for each and every file in the target container. This would have a terrible impact. Pure “replace all” strategy would be more efficient, but still have some impacts

ndeloof on Dec 13, 2023

Same problem over here. As is, the docker compose watch feature is a great direction but usable in most of our use cases. I followed reports and blog posts about this new feature closely and we are using docker compose heavily in our development team, however, it not have crossed my mind that this would not take care of syncing the initial state of the configured files and folders at container startup. My guess is that a lot of people will try to use docker compose watch as a direct replacement for bind mounts (e.g. in OSes with inefficiencies resulting from bind mounts) and will run into this very same problem. And due to the different experience that bind mounts give, a lot of users might have a hard time tracking this down.

I will try to go for a coordinated docker compose cp like @kimdcottrell suggested - my initial tests for this looked good. Thanks!

An MR with maybe a flag to decide if one wants to copy current state of the synced folder on container startup would be very nice! 👍

thomastweets on Nov 10, 2023

A temporary workaround to this is to open an additional new terminal session and run docker compose cp . app:/var/www/html after watch is running.

I wanted to get a PR out for this, tho I can’t quite determine why the calling *composeService.Copy() in Watch() doesn’t have the same behavior as the cli command above. The copy occurs, but the container doesn’t get updated. I had hoped to do something like: if the watch config action is set to sync or sync+restart and a new watcher is created, run a copy from the watch source path to the watch target.

kimdcottrell on Nov 1, 2023

In our case, dev containers images don’t include the source code, that currently is added via a volume at run-time.

Moving to watch functionality, in our case the “replace all” strategy is preferable as we don’t really need to check files inside the image.

If this replace can happen before the ENTRYPOINT is executed, that would be perfect, so the ENTRYPOINT can execute the fresh code at each restart.

massimeddu-sj on Dec 13, 2023

Unfortunately, I don’t know enough about docker internal workings and I currently do not have the time resources to allocate to learn in order to help here in any way.

I don’t technically need to “sync” only changed files, just take everything from host path definition and put it to the target location in container (I realise this is probably a gross oversimplification) and go from there. Is a comparison for every single file required?

edvordo on Oct 19, 2023