watchtower: "NO SUCH CONTAINER" after Gluetun update. How do I fix this?

Now this is a pretty sight!

time="2023-10-07T13:25:13-07:00" level=info msg="Watchtower 1.6.0"
time="2023-10-07T13:25:13-07:00" level=info msg="Using notifications: smtp"
time="2023-10-07T13:25:13-07:00" level=info msg="Checking all containers (except explicitly disabled with label)"
time="2023-10-07T13:25:13-07:00" level=info msg="Scheduling first run: 2023-10-07 13:26:53 -0700 PDT"
time="2023-10-07T13:25:13-07:00" level=info msg="Note that the first check will be performed in 1 minute, 39 seconds"
time="2023-10-07T13:27:05-07:00" level=info msg="Found new qmcgaw/gluetun:latest image (dbbf445e33b6)"
time="2023-10-07T13:27:17-07:00" level=info msg="Stopping /prowlarr (e0ab17e2520a) with SIGTERM"
time="2023-10-07T13:27:24-07:00" level=info msg="Stopping /sabnzbd (90aa13024b0e) with SIGTERM"
time="2023-10-07T13:27:29-07:00" level=info msg="Stopping /gluetun (6b41a9cabdf1) with SIGTERM"
time="2023-10-07T13:27:30-07:00" level=info msg="Creating /gluetun"
time="2023-10-07T13:27:31-07:00" level=info msg="Creating /sabnzbd"
time="2023-10-07T13:27:31-07:00" level=info msg="Creating /prowlarr"
time="2023-10-07T13:27:32-07:00" level=info msg="Removing image a86ee1e11f83"
time="2023-10-07T13:27:32-07:00" level=info msg="Session done" Failed=0 Scanned=19 Updated=1 notify=no

Thank you to everyone and the hard work that was put into resolving this issue!

Currently having the same issue after watchtower updates the gluetun container. The linked containers get stopped (not in the right order) but fail to start afterwards.

- Found new ghcr.io/linuxserver/qbittorrent:latest image (922efe500213) 
- Found new qmcgaw/gluetun:latest image (29d0ecffcccd) 
- Found new willfarrell/autoheal:latest image (c20c9c900dff) 
- Found new lscr.io/linuxserver/radarr:latest image (0f78e9253e39) 
- Stopping /radarr (ad05717e8a93) with SIGTERM 
- Stopping /autoheal (45edee9c8a60) with SIGTERM 
- Stopping /gluetun (746a51b7b07b) with SIGTERM 
- Stopping /qbittorrent (106d05adebd1) with SIGTERM 
- Stopping /prowlarr (67a0cefcbe3a) with SIGTERM 
- Creating /prowlarr Error response from daemon: No such container: 746a51b7b07bce24a327227e4b37b3cd8fb766ec72c7d3b9b2bcafddc5140521
- Creating /qbittorrent Error response from daemon: No such container: 746a51b7b07bce24a327227e4b37b3cd8fb766ec72c7d3b9b2bcafddc5140521
- Creating /gluetun
- Creating /autoheal
- Creating /radarr
- Removing image 45fbd13c4e50
- Removing image 7d66fc7c1004
- Removing image 12a3e737b691

If i can provide any info/logs or help with testing then let me know.

The PR mentioned here https://github.com/containrrr/watchtower/issues/1656#issuecomment-1615932853 has already been merged, and as mentioned in the last comment of the PR it can be tested ising latest-dev image tag

There is a PR from last year but not merged yet Networkmode service #1429

I’d redeploy the stack manually every time gluetun updates.

Did you have any luck resolving this issue?

Absolutely if you could figure out a resolution that would be tremendous. I will post my media stack compose and watchtower compose below. Disregard all the # on the media stack. It needs a good tidy-up as I was experimenting with a lot of things ahaha. This is the current state of both my media-stack and watchtower compose files.

WATCHTOWER DOCKER COMPOSE

version: "3"
services:
  watchtower:
    image: containrrr/watchtower:latest
    container_name: watchtower
    network_mode: bridge
    environment:
      - TZ=Australia/Melbourne
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_INCLUDE_RESTARTING=true
      - WATCHTOWER_POLL_INTERVAL=43200
      - WATCHTOWER_HTTP_API_METRICS=true
      - WATCHTOWER_HTTP_API_TOKEN=xxxxxxxxxxxxxxxxxxxxx
      - WATCHTOWER_LABEL_ENABLE=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - 8367:8080
    restart: unless-stopped

MEDIA-STACK DOCKER COMPOSE

version: "3"
services:
  gluetun:
    image: qmcgaw/gluetun:latest
    container_name: gluetun
    network_mode: "gluetun"
    labels:
      - com.centurylinklabs.watchtower.enable=true
    environment:
      - TZ=Australia/Melbourne
      - PUID=1000
      - PGID=1000
      - VPNSP=private internet access
      - REGION=Netherlands
      - OPENVPN_USER=xxxxxxxxxxxxxxx
      - OPENVPN_PASSWORD=xxxxxxxxxxx
      - FIREWALL_OUTBOUND_SUBNETS=192.168.0.0/24
    volumes:
      - /usb1/Configs/Gluetun:/gluetun
    ports:
#      - 9080:9080 #QBitTorrent Web-UI
#      - 6881:6881 #QBitTorrent
#      - 6881:6881/udp #QBitTorrent
      - 8084:8080 #SABnzbd
      - 9696:9696 #Prowlarr
#      - 7878:7878 #Radarr
#      - 8989:8989 #Sonarr
#      - 8686:8686 #Lidarr
#      - 8787:8787 #Readarr
#      - 6767:6767 #Bazarr
#      - 8505:5055 #Overseerr
      - 8000:8000 #API
    cap_add:
      - NET_ADMIN
    restart: unless-stopped

#  qbittorrent:
#    image: linuxserver/qbittorrent:latest
#    container_name: qbittorrent
#    network_mode: "service:gluetun"
#    environment:
#      - PUID=1000
#      - PGID=1000
#      - TZ=Australia/Melbourne
#      - UMASK=022
#      - WEBUI_PORT=9080
#    volumes:
#      - /usb1/Configs/qBittorrent:/config
#      - /usb1/Media:/Media
#    depends_on:
#      - gluetun
#    restart: unless-stopped

  sabnzbd:
    image: linuxserver/sabnzbd:latest
    container_name: sabnzbd
    network_mode: "service:gluetun"
#    network_mode: "bridge"
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Australia/Melbourne
    volumes:
      - /usb1/Configs/SABnzbd:/config
      - /usb1/Media:/Media
#    ports:
#      - 8084:8080
    depends_on:
      - gluetun
    labels:
#      - com.centurylinklabs.watchtower.enable=true
      - com.centurylinklabs.watchtower.depends-on=/gluetun
    restart: unless-stopped

  prowlarr:
    image: linuxserver/prowlarr:latest
    container_name: prowlarr
    network_mode: "service:gluetun"
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Australia/Melbourne
    volumes:
      - /usb1/Configs/Prowlarr:/config
    depends_on:
      - gluetun
    labels:
#      - com.centurylinklabs.watchtower.enable=true
      - com.centurylinklabs.watchtower.depends-on=/gluetun
    restart: unless-stopped

  radarr:
    image: linuxserver/radarr:latest
    container_name: radarr
#    network_mode: "service:gluetun"
    network_mode: "bridge"
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Australia/Melbourne
    volumes:
      - /usb1/Configs/Radarr:/config
      - /usb1/Media:/Media
    ports:
      - 7878:7878
#    depends_on:
#      - gluetun
    restart: unless-stopped

  sonarr:
    image: linuxserver/sonarr:develop
    container_name: sonarr
#    network_mode: "service:gluetun"
    network_mode: "bridge"
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Australia/Melbourne
    volumes:
      - /usb1/Configs/Sonarr:/config
      - /usb1/Media:/Media
    ports:
      - 8989:8989
#    depends_on:
#      - gluetun
    restart: unless-stopped

  lidarr:
    image: linuxserver/lidarr:latest
    container_name: lidarr
#    network_mode: "service:gluetun"
    network_mode: "bridge"
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Australia/Melbourne
    volumes:
      - /usb1/Configs/Lidarr:/config
      - /usb1/Media:/Media
    ports:
      - 8686:8686
#    depends_on:
#      - gluetun
    restart: unless-stopped

  readarr:
    image: linuxserver/readarr:develop
    container_name: readarr
#    network_mode: "service:gluetun"
    network_mode: "bridge"
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Australia/Melbourne
    volumes:
      - /usb1/Configs/Readarr:/config
      - /usb1/Media:/Media
    ports:
      - 8787:8787
#    depends_on:
#      - gluetun
    restart: unless-stopped

  bazarr:
    image: linuxserver/bazarr:latest
    container_name: bazarr
#    network_mode: "service:gluetun"
    network_mode: "bridge"
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Australia/Melbourne
    volumes:
      - /usb1/Configs/Bazarr:/config
      - /usb1/Media:/Media
    ports:
      - 6767:6767
#    depends_on:
#      - gluetun
    restart: unless-stopped

  jellyfin:
    image: jellyfin/jellyfin:latest
    container_name: jellyfin
    network_mode: "bridge"
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Australia/Melbourne
    volumes:
      - /usb1/Configs/Jellyfin:/config
      - /usb1/Media/TV-Shows:/data/tvshows
      - /usb1/Media/Movies:/data/movies
    ports:
      - 8096:8096
    restart: unless-stopped

  plex:
    image: linuxserver/plex:latest
    container_name: plex
    network_mode: host
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Australia/Melbourne
      - VERSION=docker
    volumes:
      - /usb1/Configs/Plex:/config
      - /usb1/Media/TV-Shows:/data/tvshows
      - /usb1/Media/Movies:/data/movies
    restart: unless-stopped

  overseerr:
    image: linuxserver/overseerr:latest
    container_name: overseerr
#    network_mode: "service:gluetun"
    network_mode: "bridge"
    environment:
      - PUID=1000
      - GUID=1000
      - TZ=Australia/Melbourne
    volumes:
      - /usb1/Configs/Overseerr:/app/config
    ports:
      - 8505:5055
    restart: unless-stopped
    depends_on:
#      - gluetun
      - radarr
      - sonarr

  tautulli:
    image: linuxserver/tautulli:latest
    container_name: tautulli
    network_mode: "bridge"
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Australia/Melbourne
    volumes:
      - /usb1/Configs/Tautulli:/config
    ports:
      - 8181:8181
    restart: unless-stopped

This is most likely due a duplicate of #1286, which has a PR for a purposed solution in #1429. It seems stale though, and I would like to try to make this work.

Perhaps, if you could help me by providing a full docker-compose.yml that I can use to reproduce the problem myself I can take a look?