rancher: Convoy-Gluster - Can't reuse volume name after Deleting Volume and Purging

EDIT: This is copied from a from a forum post: https://forums.rancher.com/t/cant-delete-storage-pool-volumes-and-reuse-the-volume-names/1797

1. Create a stack with a volume using convoy-gluster storage driver
2. Put some data in the volume
3. Delete the stack
4. From the Infrastructure -> Storage Pools page. Delete and then Purge the volume.
   • Volume should no longer be listed
5. Recreate the stack. Use the same volume name.
6. The container creation will fail and you might see a 500 “Couldn’t find Volume” or a 404 “no such file or directory” . Sample 500 error from the rancher server logs:

2016-02-21 00:31:13,114 ERROR [:] [] [] [] [rviceReplay-103] [i.c.p.e.e.i.ProcessEventListenerImpl] Unknown exception running process [instance.start:253938] on [10577] io.cattle.platform.eventing.exception.EventExecutionException: 500 Server Error: Internal Server Error ("Couldn't find volume.")

Sample 404 error from the rancher server logs:

2016-02-20 23:40:59,975 ERROR [c33013a0-c24d-4cb9-9a25-4851e5f68c2d:253519] [instance:10569] [instance.start] [] [rviceReplay-110] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception io.cattle.platform.eventing.exception.EventExecutionException: 404 Client Error: Not Found ("open /var/lib/rancher/convoy/convoy-gluster-f84579f3-1548-43b5-b934-f445bd6011e5/glusterfs/mounts/data01/test: no such file or directory")

There are a lot of containers involved in gluster and convoy, so I’m not sure which logs might help. Please let me know if you would like me to post other logs.

Rancher Versions: Rancher v0.59.0 Cattle v0.148.0 User Interface v0.90.0 Rancher Compose v0.7.2

Test docker-compose.yml

test:
  labels:
    io.rancher.container.pull_image: always
  tty: true
  image: busybox
  volumes:
    - test01:/test
  stdin_open: true
  volume_driver: convoy-gluster
  command: 
    - /bin/sh
    - -c
    - while true; do echo hello world; sleep 1; done

Test rancher-compose.yml

test:
  scale: 1