moby: templated volume name syntax not supported in compose v3 format?

Ok, did a little more playing around and this seems to work (though TBH feels a little far from ideal):

version: '3.1'
services:
  redis:
    image: redis
    volumes:
      - redisVol:/data
volumes:
  redisVol:
    external:
      name: '{{index .Service.Labels "com.docker.stack.namespace"}}_redisVol-{{.Task.Slot}}'

$ docker stack deploy -c docker-compose.yml redis
Creating network redis_default
Creating service redis_redis
$ docker volume ls | grep redis
local               redis_redisVol-1
$ docker exec $(docker inspect $(docker service ps redis_redis -q) -f "{{.Status.ContainerStatus.ContainerID}}") touch /data/hello_world
$ docker run -v redis_redisVol-1:/tmp alpine ls -alh /tmp/hello_world
-rw-r--r--    1 root     root           0 Feb  7 01:37 /tmp/hello_world

This is at least workable for now.

thanks @dnephin.

This doesn’t work:

version: '3.1'
services:
  redis:
    image: redis
    volumes:
      - "redisVol-{{.Task.Slot}}:/data"
volumes:
  'redisVol-{{.Task.Slot}}': {}

$ docker stack deploy -c c.yml redis
redisVol-{{.Task.Slot}} Additional property redisVol-{{.Task.Slot}} is not allowed

BUT! This works!

version: '3.1'
services:
  redis:
    image: redis
    volumes:
      - redisVol:/data
volumes:
  redisVol:
    external:
      name: 'redisVol-{{.Task.Slot}}'

$ docker stack deploy -c c.yml redis
Creating network redis_default
Creating service redis_redis
$ docker volume ls | grep redis
local               redisVol-1

I didn’t realize external volumes would be created automatically, but I guess that makes sense.

Scaling works too:

$ docker service scale redis_redis=3
redis_redis scaled to 3
$ docker volume ls | grep redis     
local               redisVol-1
local               redisVol-2
local               redisVol-3

🎉

This is really needed for scalable stateful docker ‘services’ Each task in a service should be able to specify its own persistent volume.

Since this is already supported by the docker service command line using templates for volumes, there should be some way to specify it via the docker compose YML file as well.(when deployed using docker stack deploy)

From the ‘docker for azure’ docs - https://docs.docker.com/docker-for-azure/persistent-data-volumes/#use-a-unique-volume-per-task

docker service create \
  --replicas 5 \
  --name ping2 \
  --mount type=volume,volume-driver=cloudstor:aws,source={{.Service.Name}}-{{.Task.Slot}}-vol,destination=/mydata \
  alpine ping docker.com

There is currently no way to achieve this via docker stack deploy using the compose YML file. As pointed out in the discussion it is possible to use ‘local’ volume and bind mounts but not custom volume drivers like rexray/ebd or cloudstor:aws etc. Looks like a bug to me.

This works with bind mounts which is useful for people using NFS/etc:

version: '3.2'
services:
  hello:
    image: alpine
    hostname: 'hello-{{.Task.Slot}}'
    environment:
      SLOT: '{{.Task.Slot}}'
      SERVICE: '{{.Service.Name}}'
      STACK: '{{index .Service.Labels "com.docker.stack.namespace"}}'
    command:
    - sh
    - -c
    - |
      echo hi! im $${HOSTNAME}, task-$${SLOT} of $${SERVICE} in stack: $${STACK}
      ls -l /data
      sleep 1000
    volumes:
    - type: bind
      source: ./{{.Task.Slot}}
      target: /data
    deploy:
      replicas: 3

mydir {1..3}
touch 1/a 2/b 3/c
docker deploy -c stack.yml mounts
docker service logs -f mounts_hello

mounts_hello.3.7hlgvxpla3f6@moby    | hi! im hello-3, task-3 of mounts_hello in stack: mounts
mounts_hello.1.cxkdeu4jhg1n@moby    | hi! im hello-1, task-1 of mounts_hello in stack: mounts
mounts_hello.2.x7yt0o0g5he5@moby    | hi! im hello-2, task-2 of mounts_hello in stack: mounts
mounts_hello.2.x7yt0o0g5he5@moby    | total 0
mounts_hello.3.7hlgvxpla3f6@moby    | total 0
mounts_hello.1.cxkdeu4jhg1n@moby    | total 0
mounts_hello.2.x7yt0o0g5he5@moby    | -rwxr-xr-x    1 root     root             0 Jun 22 05:29 b
mounts_hello.1.cxkdeu4jhg1n@moby    | -rwxr-xr-x    1 root     root             0 Jun 22 05:29 a
mounts_hello.3.7hlgvxpla3f6@moby    | -rwxr-xr-x    1 root     root             0 Jun 22 05:29 c

The older bind mount syntax also works. (unlike volume-name in this issue)

ex:

services:
  hello:
    volumes:
    - ./{{.Task.Slot}}:/data

Docker 20.10 finally gave us hostnames that resolve.

So deployments like this, that used to require 4 services, are now almost possible :-

services:
  minio:
    image: "minio/minio"
    hostname: "minio{{.Task.Slot}}"
    command: "server http://minio{1...4}/data"
    environment:
      MINIO_ACCESS_KEY: "minio_access_key"
      MINIO_SECRET_KEY: "minio_secret_key"
    volumes:
    - "disk{{.Task.Slot}}:/data"
    deploy:
      replicas: 4
volumes:
  disk1:
  disk2:
  disk3:
  disk4:

@hairyhenderson I cannot find any documentation regarding this feature. Can you clarify how you came across this?

I was about to close this, but then I realized that this causes a problem if I want to deploy the stack multiple times with different stack names (because the volume is external).

I suppose for this to work as expected it’d have to be possible to set the name of the volume separately from the key:

volumes:
  redisVol:
    name: redisVol-{{.Task.Slot}}
    driver: local

I’d then expect a volume to be created named <stack name>_redisVol-1

(obviously, this doesn’t work)

@chrisbecke Great! When you say almost possible, what is still missing? I haven’t tried this in a while, but you should be able to combine the volume definitions into one (disk1…disk4 to just disk) if you put the {{.Task.Slot}} into the volume name there. command should also support it.

When using external, the volume is expected to be created up-front (“external”). Having .Task.Slot in the template for an external volume won’t work, because that information is not available for a global “volume” definition, but only for a “task” that uses the volume.

The volume name under services in a Compose file is not the actual volume name. It’s a reference to a volume defined in the top level volumes section of the Compose file.

It might work if you add:

volumes:
  'redisVol-{{.Task.Slot}}': {}

Otherwise you might have to use external volumes:

volumes:
  redisVol:
    external:
      name: 'redisVol-{{.Task.Slot}}'