kubernetes: Unit Test k8s.io/kubernetes/cmd/genkubedocs fails in ppc64le

Which jobs are failing:

https://prow.k8s.io/job-history/gs/ppc64le-kubernetes/logs/periodic-kubernetes-unit-test-ppc64le

Which test(s) are failing:

k8s.io/kubernetes/cmd/genkubedocs

Since when has it been failing:

CI job passed when commit ID was 0df8c6973 Next iteration after 3 hrs interval failed. The commit ID was faa3a5fbd

Testgrid link:

https://k8s-testgrid.appspot.com/sig-node-ppc64le#unit-tests

Reason for failure:

Console from CI job failure

unexpected fault address 0xffffffffffff7990
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x3 addr=0xffffffffffff7990 pc=0x152bb21c]

goroutine 1 [running, locked to thread]:
runtime.throw(0x16fb4fcf, 0x5)
        /usr/local/go/src/runtime/panic.go:1117 +0x5c fp=0xc0007e34d0 sp=0xc0007e3490 pc=0x11aefb3c
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:741 +0x274 fp=0xc0007e3518 sp=0xc0007e34d0 pc=0x11b09894
k8s.io/kubernetes/vendor/github.com/Azure/go-autorest/autorest/azure.init()
        /root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/Azure/go-autorest/autorest/azure/environments.go:34 +0x260 fp=0xc0007e35b0 sp=0xc0007e3538 pc=0x13c4ce80
runtime.doInit(0x1a0b2020)
        /usr/local/go/src/runtime/proc.go:6265 +0xf0 fp=0xc0007e3710 sp=0xc0007e35b0 pc=0x11b00b90
runtime.doInit(0x1a0d89e0)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc0007e3870 sp=0xc0007e3710 pc=0x11b00b04
runtime.doInit(0x1a08b4e0)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc0007e39d0 sp=0xc0007e3870 pc=0x11b00b04
runtime.doInit(0x1a0c19c0)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc0007e3b30 sp=0xc0007e39d0 pc=0x11b00b04
runtime.doInit(0x1a0d4a60)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc0007e3c90 sp=0xc0007e3b30 pc=0x11b00b04
runtime.doInit(0x1a0b1300)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc0007e3df0 sp=0xc0007e3c90 pc=0x11b00b04
runtime.doInit(0x1a08dae0)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc0007e3f50 sp=0xc0007e3df0 pc=0x11b00b04
runtime.main()
        /usr/local/go/src/runtime/proc.go:208 +0x214 fp=0xc0007e3fc0 sp=0xc0007e3f50 pc=0x11af21a4
runtime.goexit()
        /usr/local/go/src/runtime/asm_ppc64x.s:869 +0x4 fp=0xc0007e3fc0 sp=0xc0007e3fc0 pc=0x11b29a54

goroutine 4 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x1a198f60)
        /root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1169 +0x90
created by k8s.io/kubernetes/vendor/k8s.io/klog/v2.init.0
        /root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:417 +0x1a8
FAIL    k8s.io/kubernetes/cmd/genkubedocs       0.582s
FAIL
make: *** [Makefile:184: test] Error 1
[root@rajalakshmi-workspace kubernetes]#

Anything else we need to know:

Fails only in ppc64le architecture. Passes in amd64.

ppc64le architecture:

[root@rajalakshmi-workspace ~]# go version
go version go1.16 linux/ppc64le
[root@rajalakshmi-workspace ~]# export KUBE_TIMEOUT='--timeout=600s'
[root@rajalakshmi-workspace ~]# export KUBE_KEEP_VERBOSE_TEST_OUTPUT=y
[root@rajalakshmi-workspace ~]# export LOG_LEVEL=4
[root@rajalakshmi-workspace ~]# rm -rf kubernetes
[root@rajalakshmi-workspace ~]# git clone https://github.com/kubernetes/kubernetes
Cloning into 'kubernetes'...
remote: Enumerating objects: 190, done.
remote: Counting objects: 100% (190/190), done.
remote: Compressing objects: 100% (125/125), done.
remote: Total 1208544 (delta 87), reused 65 (delta 65), pack-reused 1208354
Receiving objects: 100% (1208544/1208544), 729.04 MiB | 27.02 MiB/s, done.
Resolving deltas: 100% (869523/869523), done.
Checking out files: 100% (20437/20437), done.
[root@rajalakshmi-workspace ~]# cd kubernetes
[root@rajalakshmi-workspace kubernetes]# make test KUBE_RACE=-race WHAT=./cmd/genkubedocs
+++ [0311 08:46:48] Building go targets for linux/ppc64le:
    ./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen
Generating prerelease lifecycle code for 27 targets
+++ [0311 08:46:54] Building go targets for linux/ppc64le:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
Generating deepcopy code for 227 targets
+++ [0311 08:47:05] Building go targets for linux/ppc64le:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
Generating defaulter code for 89 targets
+++ [0311 08:47:18] Building go targets for linux/ppc64le:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
Generating conversion code for 124 targets
+++ [0311 08:47:45] Building go targets for linux/ppc64le:
    ./vendor/k8s.io/kube-openapi/cmd/openapi-gen
Generating openapi code for KUBE
Generating openapi code for AGGREGATOR
Generating openapi code for APIEXTENSIONS
Generating openapi code for CODEGEN
Generating openapi code for SAMPLEAPISERVER
+++ [0311 08:48:03] Building go targets for linux/ppc64le:
    ./vendor/github.com/go-bindata/go-bindata/go-bindata
+++ [0311 08:48:04] Running tests without code coverage
unexpected fault address 0xffffffffffff7990
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x3 addr=0xffffffffffff7990 pc=0x152bb21c]

goroutine 1 [running, locked to thread]:
runtime.throw(0x16fb4fcf, 0x5)
        /usr/local/go/src/runtime/panic.go:1117 +0x5c fp=0xc0007e34d0 sp=0xc0007e3490 pc=0x11aefb3c
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:741 +0x274 fp=0xc0007e3518 sp=0xc0007e34d0 pc=0x11b09894
k8s.io/kubernetes/vendor/github.com/Azure/go-autorest/autorest/azure.init()
        /root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/Azure/go-autorest/autorest/azure/environments.go:34 +0x260 fp=0xc0007e35b0 sp=0xc0007e3538 pc=0x13c4ce80
runtime.doInit(0x1a0b2020)
        /usr/local/go/src/runtime/proc.go:6265 +0xf0 fp=0xc0007e3710 sp=0xc0007e35b0 pc=0x11b00b90
runtime.doInit(0x1a0d89e0)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc0007e3870 sp=0xc0007e3710 pc=0x11b00b04
runtime.doInit(0x1a08b4e0)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc0007e39d0 sp=0xc0007e3870 pc=0x11b00b04
runtime.doInit(0x1a0c19c0)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc0007e3b30 sp=0xc0007e39d0 pc=0x11b00b04
runtime.doInit(0x1a0d4a60)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc0007e3c90 sp=0xc0007e3b30 pc=0x11b00b04
runtime.doInit(0x1a0b1300)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc0007e3df0 sp=0xc0007e3c90 pc=0x11b00b04
runtime.doInit(0x1a08dae0)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc0007e3f50 sp=0xc0007e3df0 pc=0x11b00b04
runtime.main()
        /usr/local/go/src/runtime/proc.go:208 +0x214 fp=0xc0007e3fc0 sp=0xc0007e3f50 pc=0x11af21a4
runtime.goexit()
        /usr/local/go/src/runtime/asm_ppc64x.s:869 +0x4 fp=0xc0007e3fc0 sp=0xc0007e3fc0 pc=0x11b29a54

goroutine 4 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x1a198f60)
        /root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1169 +0x90
created by k8s.io/kubernetes/vendor/k8s.io/klog/v2.init.0
        /root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:417 +0x1a8
FAIL    k8s.io/kubernetes/cmd/genkubedocs       0.582s
FAIL
make: *** [Makefile:184: test] Error 1
[root@rajalakshmi-workspace kubernetes]# uname -a
Linux rajalakshmi-workspace 4.18.0-193.19.1.el8_2.ppc64le #1 SMP Mon Sep 14 14:48:04 UTC 2020 ppc64le ppc64le ppc64le GNU/Linux
[root@rajalakshmi-workspace kubernetes]#

amd64 architecture:

root@ragini-x861:~# go version
go version go1.16 linux/amd64
root@ragini-x861:~# export KUBE_TIMEOUT='--timeout=600s'
root@ragini-x861:~# export KUBE_KEEP_VERBOSE_TEST_OUTPUT=y
root@ragini-x861:~# export LOG_LEVEL=4
root@ragini-x861:~# rm -rf kubernetes
root@ragini-x861:~#  git clone https://github.com/kubernetes/kubernetes
Cloning into 'kubernetes'...
remote: Enumerating objects: 191, done.
remote: Counting objects: 100% (191/191), done.
remote: Compressing objects: 100% (127/127), done.
remote: Total 1208544 (delta 88), reused 64 (delta 64), pack-reused 1208353
Receiving objects: 100% (1208544/1208544), 729.01 MiB | 4.58 MiB/s, done.
Resolving deltas: 100% (869561/869561), done.
Checking out files: 100% (20437/20437), done.
root@ragini-x861:~# cd kubernetes/
root@ragini-x861:~/kubernetes# make test KUBE_RACE=-race WHAT=./cmd/genkubedocs
+++ [0311 01:04:50] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen
Generating prerelease lifecycle code for 27 targets
+++ [0311 01:04:57] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
Generating deepcopy code for 227 targets
+++ [0311 01:05:11] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
Generating defaulter code for 89 targets
+++ [0311 01:05:26] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
Generating conversion code for 124 targets
+++ [0311 01:05:54] Building go targets for linux/amd64:
    ./vendor/k8s.io/kube-openapi/cmd/openapi-gen
Generating openapi code for KUBE
Generating openapi code for AGGREGATOR
Generating openapi code for APIEXTENSIONS
Generating openapi code for CODEGEN
Generating openapi code for SAMPLEAPISERVER
+++ [0311 01:06:14] Building go targets for linux/amd64:
    ./vendor/github.com/go-bindata/go-bindata/go-bindata
+++ [0311 01:06:16] Running tests without code coverage
ok      k8s.io/kubernetes/cmd/genkubedocs       2.507s
root@ragini-x861:~/kubernetes# uname -a
Linux ragini-x861.fyre.ibm.com 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
root@ragini-x861:~/kubernetes#

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (16 by maintainers)

Most upvoted comments

The test passed withe new gov version: go version go1.16.5 linux/ppc64le

There is a golang update task here - https://github.com/kubernetes/release/issues/2107, hope things will be fine post this fix.

+1
mkumatag on Jun 8, 2021

I’m seeing the latest go 1.16 version as: go version go1.16.5 linux/ppc64le

Has this been tried with the most recent?

The test passed withe new gov version: go version go1.16.5 linux/ppc64le

[root@karthik-workspace ~]# go version go version go1.16.5 linux/ppc64le [root@karthik-workspace ~]# export KUBE_TIMEOUT=‘–timeout=600s’ [root@karthik-workspace ~]# export KUBE_KEEP_VERBOSE_TEST_OUTPUT=y [root@karthik-workspace ~]# export LOG_LEVEL=4 [root@karthik-workspace ~]# rm -rf kubernetes [root@karthik-workspace ~]# git clone https://github.com/kubernetes/kubernetes Cloning into ‘kubernetes’… remote: Enumerating objects: 1234983, done. remote: Counting objects: 100% (230/230), done. remote: Compressing objects: 100% (152/152), done. remote: Total 1234983 (delta 99), reused 99 (delta 78), pack-reused 1234753 Receiving objects: 100% (1234983/1234983), 749.94 MiB | 8.17 MiB/s, done. Resolving deltas: 100% (889292/889292), done. Updating files: 100% (20682/20682), done. [root@karthik-workspace ~]# make test KUBE_RACE=-race WHAT=./cmd/genkubedocs make: *** No rule to make target ‘test’. Stop. [root@karthik-workspace ~]# cd kubernetes/ [root@karthik-workspace kubernetes]# make test KUBE_RACE=-race WHAT=./cmd/genkubedocs +++ [0608 06:35:36] Building go targets for linux/ppc64le: ./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen Generating prerelease lifecycle code for 27 targets +++ [0608 06:35:49] Building go targets for linux/ppc64le: ./vendor/k8s.io/code-generator/cmd/deepcopy-gen Generating deepcopy code for 227 targets +++ [0608 06:35:59] Building go targets for linux/ppc64le: ./vendor/k8s.io/code-generator/cmd/defaulter-gen Generating defaulter code for 89 targets +++ [0608 06:36:12] Building go targets for linux/ppc64le: ./vendor/k8s.io/code-generator/cmd/conversion-gen Generating conversion code for 124 targets +++ [0608 06:36:39] Building go targets for linux/ppc64le: ./vendor/k8s.io/kube-openapi/cmd/openapi-gen Generating openapi code for KUBE Generating openapi code for AGGREGATOR Generating openapi code for APIEXTENSIONS Generating openapi code for CODEGEN Generating openapi code for SAMPLEAPISERVER +++ [0608 06:37:01] Building go targets for linux/ppc64le: ./vendor/github.com/go-bindata/go-bindata/go-bindata +++ [0608 06:37:03] Running tests without code coverage ok k8s.io/kubernetes/cmd/genkubedocs 0.386s

+1
Karthik-K-N on Jun 8, 2021
Read more comments on GitHub
← kubernetes: unidirectional UDP traffic to a NodePort service is not handled properly
kubernetes: unit tests should not require root →