rancher: Convoy-Gluster - Can't reuse volume name after Deleting Volume and Purging

EDIT: This is copied from a from a forum post: https://forums.rancher.com/t/cant-delete-storage-pool-volumes-and-reuse-the-volume-names/1797

  1. Create a stack with a volume using convoy-gluster storage driver
  2. Put some data in the volume
  3. Delete the stack
  4. From the Infrastructure -> Storage Pools page. Delete and then Purge the volume.
    • Volume should no longer be listed
  5. Recreate the stack. Use the same volume name.
  6. The container creation will fail and you might see a 500 “Couldn’t find Volume” or a 404 “no such file or directory” . Sample 500 error from the rancher server logs:
2016-02-21 00:31:13,114 ERROR [:] [] [] [] [rviceReplay-103] [i.c.p.e.e.i.ProcessEventListenerImpl] Unknown exception running process [instance.start:253938] on [10577] io.cattle.platform.eventing.exception.EventExecutionException: 500 Server Error: Internal Server Error ("Couldn't find volume.")

Sample 404 error from the rancher server logs:

2016-02-20 23:40:59,975 ERROR [c33013a0-c24d-4cb9-9a25-4851e5f68c2d:253519] [instance:10569] [instance.start] [] [rviceReplay-110] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception io.cattle.platform.eventing.exception.EventExecutionException: 404 Client Error: Not Found ("open /var/lib/rancher/convoy/convoy-gluster-f84579f3-1548-43b5-b934-f445bd6011e5/glusterfs/mounts/data01/test: no such file or directory")

There are a lot of containers involved in gluster and convoy, so I’m not sure which logs might help. Please let me know if you would like me to post other logs.

Rancher Versions: Rancher v0.59.0 Cattle v0.148.0 User Interface v0.90.0 Rancher Compose v0.7.2

Test docker-compose.yml

test:
  labels:
    io.rancher.container.pull_image: always
  tty: true
  image: busybox
  volumes:
    - test01:/test
  stdin_open: true
  volume_driver: convoy-gluster
  command: 
    - /bin/sh
    - -c
    - while true; do echo hello world; sleep 1; done

Test rancher-compose.yml

test:
  scale: 1

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 18 (7 by maintainers)

Most upvoted comments

Hi @deniseschannon, I’m use Convoy-Gluster and have many troubles because this issue. There are any chance to this issue be included in future version?

similar problem here except in my case the volumes won’t delete from the storage pool. The volume is not mounted anywhere. This is happening for every volume I delete. Using Convoy-NFS from catalog. Rancher 1.0.1.

4/29/2016 4:25:59 PMtime="2016-04-29T23:25:59Z" level=error msg="Error processing event" err="Cannot delete volume 1711. Name: gl-var. Error: Cannot delete volume gl-var. It is still mounted\n" eventId=eb1cfa5d-ee7f-4fc2-b2f1-649d476c885a eventName=storage.volume.remove resourceId=1vspm901