kubernetes: gce-ipvs CI job failures

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

There are some failures in ubuntu-gce-ipvs CI job. See, http://k8s-testgrid.appspot.com/google-gce#ubuntu-gce-ipvs

What you expected to happen:

We want to see CI job green!!

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

@Lion-Wei @stewart-yu in my team will together take care of gci-gce-ipvs CI job.

Environment:

Kubernetes version (use kubectl version):
Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

/sig network

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 30 (25 by maintainers)

Most upvoted comments

@Lion-Wei Just checked GKE Ubuntu OS image, and it has ip_vs_* modules by default.

# modprobe ip_vs_rr
# modprobe ip_vs_wrr
# modprobe ip_vs_sh

# lsmod | grep ip_vs
ip_vs_sh               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  0
ip_vs                 147456  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
libcrc32c              16384  2 raid456,ip_vs

If you want to run your test on GCE, you can use that image. There are a plenty of test jobs using Ubuntu image at https://github.com/kubernetes/test-infra/blob/master/prow/config.yaml, you can start from there.

dchen1107 on Nov 13, 2017

Yes, COS image doesn’t have ip_vs_* modules because it never being requested. We requested a couple of days ago, and they are working on it. But it will take some time to release the changes.

Meanwhile, since the feature was only enabled for a particular test job, you can build your own OS image, and override the default OS image for your test job.

dchen1107 on Nov 10, 2017

@cmluciano you did: https://github.com/kubernetes/kubernetes/pull/46580#issuecomment-342980714

BenTheElder on Nov 9, 2017

@m1093782566 BTW, actually I am not sure if all kernel modules like ip_vs_rr, ip_vs_wrr, ip_vs_sh are strictly required by the IPVS proxier. If they are required, maybe we should suggest Container-Optimized OS to compile those kernel modules.

vfreex on Nov 8, 2017

@m1093782566 I guess the required kernel modules are not compiled in GCI’s distibution. Can we get the kernel compilation config file of installed kernel and take a look?

vfreex on Nov 8, 2017

@Lion-Wei Seems like all of ip_vs_rr, ip_vs_wrr, ip_vs_sh are not available in the default GCE enviroment. Here is what I got from one of the GCI nodes:

# pwd
/lib/modules/4.4.64+/kernel/net/netfilter/ipvs
# ls
ip_vs.ko
# cat /etc/os-release
BUILD_ID=9592.90.0
NAME="Container-Optimized OS"
KERNEL_COMMIT_ID=c0dda75c2c06905ff4a2e8b41de27d4f2d773856
GOOGLE_CRASH_ID=Lakitu
VERSION_ID=60
BUG_REPORT_URL=https://crbug.com/new
PRETTY_NAME="Container-Optimized OS from Google"
VERSION=60
GOOGLE_METRICS_PRODUCT_ID=26
HOME_URL="https://cloud.google.com/compute/docs/containers/vm-image/"
ID=cos

Adding @kubernetes/sig-node-bugs for more insights. Why aren’t those common IPVS modules included in GCI image?

MrHohn on Nov 8, 2017

https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ipvs/ Seems like tests are passing after https://github.com/kubernetes/kubernetes/pull/55004 went in.

MrHohn on Nov 2, 2017

/assign

adding myself for tracking purposes

cmluciano on Nov 2, 2017