kubernetes: One of the flexvolume driver added error will block the in coming flexvolume
Is this a BUG REPORT or FEATURE REQUEST?:
Uncomment only one, leave it on its own line:
/kind bug
What happened:
I add multi flexvolume for testing, but some of them return error, I continue add new available flexvolume after the bad flexvolume, but the kubelet will continually through error:
MESSAGE: "E0302 06:55:03.630491 30575 desired_state_of_world_populator.go:288] Failed to add volume "2f2test" (specName: "2f2test") for pod "9c56e9b7-1de6-11e8-aabc-42010a8001f8" to desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "2f2test" err=no volume plugin matched"
Here is the error when add the bad flexvolume driver.
MESSAGE: "E0302 05:13:49.243751 30575 plugins.go:515] Error dynamically probing plugins: [Error creating Flexvolume plugin from directory rancher.io~longhorn, skipping. Error: unexpected end of JSON input, Error creating Flexvolume plugin from directory rancher~rancherflex, skipping. Error: unexpected end of JSON input]"
Here may be the related cause, even if the probe() will continue get all latest flexvolume driver, but in the refreshProbedPlugins() it return when error occurred, so the flexvolume will never get updating.
What you expected to happen: Flexvolume should not block each other How to reproduce it (as minimally and precisely as possible):
- create a bad return flexvolume driver, for example with bad return in init function
- create a good flexvolume driver
- create a pod use the second one as volume driver
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version
): Client Version: version.Info{Major:“1”, Minor:“9”, GitVersion:“v1.9.0”, GitCommit:“925c127ec6b946659ad0fd596fa959be43f0cc05”, GitTreeState:“clean”, BuildDate:“2017-12-15T21:07:38Z”, GoVersion:“go1.9.2”, Compiler:“gc”, Platform:“darwin/amd64”} Server Version: version.Info{Major:“1”, Minor:“8+”, GitVersion:“v1.8.7-gke.1”, GitCommit:“192ccad06d24af9828cbf42330e1d915cb586406”, GitTreeState:“clean”, BuildDate:“2018-01-31T21:39:04Z”, GoVersion:“go1.8.3b4”, Compiler:“gc”, Platform:“linux/amd64”} - Cloud provider or hardware configuration: GKE
- OS (e.g. from /etc/os-release): NAME=“Ubuntu” VERSION=“16.04.3 LTS (Xenial Xerus)” ID=ubuntu ID_LIKE=debian PRETTY_NAME=“Ubuntu 16.04.3 LTS” VERSION_ID=“16.04” HOME_URL=“http://www.ubuntu.com/” SUPPORT_URL=“http://help.ubuntu.com/” BUG_REPORT_URL=“http://bugs.launchpad.net/ubuntu/” VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial
- Kernel (e.g.
uname -a
): Linux gke-michelia-flextvolume-default-pool-dbaa5b9c-vdfg 4.13.0-1007-gcp #10-Ubuntu SMP Fri Jan 12 13:56:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux - Install tools: GKE
- Others:
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 16 (3 by maintainers)
@verult @linyouchong Could you take a look at this issue? Rook users continue to be impacted by flex drivers failing to load consistently. This has been an issue since K8s v1.11. See @jbw976’s analysis of the issue here. Thanks!