rancher: docker stops responding when convoy-gluster stack is deleted when volumes using convoy-gluster are still present.

Rancher Version: v1.0.1-rc6

Docker version: v1.10.3

I dont have the exact steps that leads to this problem. Following are the steps I had been doing before I got to a state where containers failed to start in one of the hosts and is stuck in “networking” state.

Created convoy-gluster and glusterfs stack. Created few services using volume-drivers set to “convoy-gluster”. Deleted these services. Created convoy-nfs stack. Created few services using volume-drivers set to “convoy-nfs”. Deleted these services.

Deleted convoy-gluster , glusterfs stack and convoy-nfs stack.

When i repeated the above set of tests couple of times is when I get to a state where I see instances deployed in one of the hosts gets stuck in “networking” state because of docker process hanging.

docker.logs:

ESC[31mERROESC[0m[0166] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0166] Handler for GET /v1.21/volumes/cb472e1a150488644d190baa04d2423e3f1df67129d3df4864a459601e32ec76 returned error: get cb472e1a150488644d190baa04d2423e3f1df67129d3df4864a459601e32ec76: no such volume 
ESC[31mERROESC[0m[0166] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0166] Handler for GET /v1.21/volumes/87a5c8424446bb505006c91af6dca75a4744609636a0e1058c7fe91f98d87c27 returned error: get 87a5c8424446bb505006c91af6dca75a4744609636a0e1058c7fe91f98d87c27: no such volume 
ESC[31mERROESC[0m[0166] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0166] Handler for GET /v1.21/volumes/92594423d573782225801ac7615f7354b41188b7e20f5575140f4781a2201e63 returned error: get 92594423d573782225801ac7615f7354b41188b7e20f5575140f4781a2201e63: no such volume 
ESC[31mERROESC[0m[0166] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0166] Handler for GET /v1.21/volumes/f62ac358b704c60bdbb757cd8a81e873c5363542cbc1e2227e20c62706c1ef10 returned error: get f62ac358b704c60bdbb757cd8a81e873c5363542cbc1e2227e20c62706c1ef10: no such volume 
ESC[31mERROESC[0m[0166] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0166] Handler for GET /v1.21/volumes/cb472e1a150488644d190baa04d2423e3f1df67129d3df4864a459601e32ec76 returned error: get cb472e1a150488644d190baa04d2423e3f1df67129d3df4864a459601e32ec76: no such volume 
ESC[31mERROESC[0m[0166] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0166] Handler for GET /v1.21/volumes/da107d35670d67178a79a1bf8cedbb6fab3728edc44ce85de4324b671692313a returned error: get da107d35670d67178a79a1bf8cedbb6fab3728edc44ce85de4324b671692313a: no such volume 
ESC[31mERROESC[0m[0166] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0166] Handler for GET /v1.21/volumes/87a5c8424446bb505006c91af6dca75a4744609636a0e1058c7fe91f98d87c27 returned error: get 87a5c8424446bb505006c91af6dca75a4744609636a0e1058c7fe91f98d87c27: no such volume 
ESC[31mERROESC[0m[0166] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0166] Handler for GET /v1.21/volumes/92594423d573782225801ac7615f7354b41188b7e20f5575140f4781a2201e63 returned error: get 92594423d573782225801ac7615f7354b41188b7e20f5575140f4781a2201e63: no such volume 
ESC[31mERROESC[0m[0196] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0196] Handler for GET /v1.21/volumes/590b322cb4ceb33cbec993ce491e1d03f4e44c8a42d60776f040d62a6cd38479 returned error: get 590b322cb4ceb33cbec993ce491e1d03f4e44c8a42d60776f040d62a6cd38479: no such volume 
ESC[31mERROESC[0m[0196] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0196] Handler for GET /v1.21/volumes/f62ac358b704c60bdbb757cd8a81e873c5363542cbc1e2227e20c62706c1ef10 returned error: get f62ac358b704c60bdbb757cd8a81e873c5363542cbc1e2227e20c62706c1ef10: no such volume 
ESC[31mERROESC[0m[0196] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0196] Handler for GET /v1.21/volumes/cb472e1a150488644d190baa04d2423e3f1df67129d3df4864a459601e32ec76 returned error: get cb472e1a150488644d190baa04d2423e3f1df67129d3df4864a459601e32ec76: no such volume 
ESC[31mERROESC[0m[0197] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0197] Handler for GET /v1.21/volumes/590b322cb4ceb33cbec993ce491e1d03f4e44c8a42d60776f040d62a6cd38479 returned error: get 590b322cb4ceb33cbec993ce491e1d03f4e44c8a42d60776f040d62a6cd38479: no such volume 
ESC[31mERROESC[0m[0197] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0197] Handler for GET /v1.21/volumes/cb472e1a150488644d190baa04d2423e3f1df67129d3df4864a459601e32ec76 returned error: get cb472e1a150488644d190baa04d2423e3f1df67129d3df4864a459601e32ec76: no such volume 
ESC[31mERROESC[0m[0197] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0197] Handler for GET /v1.21/volumes/f62ac358b704c60bdbb757cd8a81e873c5363542cbc1e2227e20c62706c1ef10 returned error: get f62ac358b704c60bdbb757cd8a81e873c5363542cbc1e2227e20c62706c1ef10: no such volume 
ESC[31mERROESC[0m[0211] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0211] Handler for GET /v1.21/volumes/da107d35670d67178a79a1bf8cedbb6fab3728edc44ce85de4324b671692313a returned error: get da107d35670d67178a79a1bf8cedbb6fab3728edc44ce85de4324b671692313a: no such volume 
ESC[31mERROESC[0m[0211] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0211] Handler for GET /v1.21/volumes/da107d35670d67178a79a1bf8cedbb6fab3728edc44ce85de4324b671692313a returned error: get da107d35670d67178a79a1bf8cedbb6fab3728edc44ce85de4324b671692313a: no such volume 
ESC[31mERROESC[0m[0226] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0226] Handler for GET /v1.21/volumes/061be1c603257146d356e8691e9826474a0202b23fa9d2598f78a3511c998561 returned error: get 061be1c603257146d356e8691e9826474a0202b23fa9d2598f78a3511c998561: no such volume 
ESC[31mERROESC[0m[0226] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0226] Handler for GET /v1.21/volumes/061be1c603257146d356e8691e9826474a0202b23fa9d2598f78a3511c998561 returned error: get 061be1c603257146d356e8691e9826474a0202b23fa9d2598f78a3511c998561: no such volume 
ESC[31mERROESC[0m[0229] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0229] Handler for GET /v1.21/volumes/test-205640 returned error: get test-205640: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0229] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0229] Handler for GET /v1.21/volumes/test-205640 returned error: get test-205640: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0306] Handler for GET /v1.18/containers/9889a3a98d763e031c6cf6957ed15d5b2d24e93594ff7a6302c38b662a1a950e/json returned error: No such container: 9889a3a98d763e031c6cf6957ed15d5b2d24e93594ff7a6302c38b662a1a950e 
ESC[31mERROESC[0m[0319] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0319] Handler for GET /v1.21/volumes/test-230556 returned error: get test-230556: no such volume 
ESC[31mERROESC[0m[0319] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0319] Handler for GET /v1.21/volumes/test-20533 returned error: get test-20533: no such volume 
ESC[31mERROESC[0m[0319] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0319] Handler for GET /v1.21/volumes/test-230556 returned error: get test-230556: no such volume 
ESC[31mERROESC[0m[0319] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0319] Handler for GET /v1.21/volumes/test-20533 returned error: get test-20533: no such volume 
ESC[31mERROESC[0m[0366] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0366] Handler for GET /v1.21/volumes/test-965480 returned error: get test-965480: no such volume 
ESC[31mERROESC[0m[0366] VolumeDriver.Get: error reading plugin resp: invalid character 'H' looking for beginning of value 
ESC[31mERROESC[0m[0366] Handler for GET /v1.21/volumes/test-965480 returned error: get test-965480: no such volume 
ESC[33mWARNESC[0m[0496] wait: no child processes                     
ESC[33mWARNESC[0m[0496] wait: no child processes                     
ESC[33mWARNESC[0m[0496] wait: no child processes                     
ESC[33mWARNESC[0m[0496] wait: no child processes                     
ESC[33mWARNESC[0m[0496] wait: no child processes                     
ESC[33mWARNESC[0m[0496] wait: no child processes                     
ESC[33mWARNESC[0m[0499] wait: no child processes                     
ESC[33mWARNESC[0m[0499] wait: no child processes                     
ESC[33mWARNESC[0m[0499] wait: no child processes                     
ESC[33mWARNESC[0m[0499] wait: no child processes                     
ESC[33mWARNESC[0m[0499] wait: no child processes                     
ESC[33mWARNESC[0m[0499] wait: no child processes                     
ESC[31mERROESC[0m[0550] Handler for GET /v1.18/containers/r-convoy-nfs_convoy-nfs-storagepool_1/json returned error: No such container: r-convoy-nfs_convoy-nfs-storagepool_1 
ESC[31mERROESC[0m[0559] Handler for GET /v1.18/containers/r-test829815_test113838_1/json returned error: No such container: r-test829815_test113838_1 

once I delete convoy-gluster.spec from /etc/docker/plugins and restart docker , docker starts responding and container deploy successfully on that host.