kubevirt: make cluster-sync not working.

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

after make cluster-up make cluster-sync fails .

The images are not build and hence while pushing images to local registry it fails.

What you expected to happen:

Images should be build and visible in host machine docker images output

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?: Log

$ make cluster-sync                                                                                                                                                                            
PUSH_LOG_FILE= ./hack/cluster-build.sh                                                                                                                                                         
Building ...                                                                                                                                                                                   
[+] Building 0.2s (9/9) FINISHED                                                                                                                                                               
 => [internal] load .dockerignore                                                                                                                                                         0.0s 
 => => transferring context: 2B                                                                                                                                                           0.0s 
 => [internal] load build definition from Dockerfile                                                                                                                                      0.0s 
 => => transferring dockerfile: 38B                                                                                                                                                       0.0s 
 => [internal] load metadata for docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12                                                       0.0s 
 => [1/4] FROM docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12                                                                         0.0s 
 => => resolve docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12                                                                         0.0s 
 => [internal] load build context                                                                                                                                                         0.0s 
 => => transferring context: 33B                                                                                                                                                          0.0s 
 => CACHED [2/4] RUN     mkdir -p /go &&     source /etc/profile.d/gimme.sh &&     go get github.com/mattn/goveralls &&     go get -u github.com/golang/mock/gomock &&     go get -u git  0.0s 
 => CACHED [3/4] RUN pip install j2cli && pip3 install operator-courier==1.3.0                                                                                                            0.0s 
 => CACHED [4/4] COPY rsyncd.conf /etc/rsyncd.conf                                                                                                                                        0.0s 
 => exporting to image                                                                                                                                                                    0.0s 
 => => exporting layers                                                                                                                                                                   0.0s 
 => => writing image sha256:6e512c03eb7b69fd0cca43aa2266799c72fead3ffa535c0390cfd7ae493d0ac8                                                                                              0.0s 
 => => naming to docker.io/library/kubevirt                                                                                                                                               0.0s 
go version go1.11.5 linux/amd64 

go version go1.11.5 linux/amd64                                                                                                                                                                
INFO: Build option --define has changed, discarding analysis cache.                                                                                                                            
INFO: Analysed 54 targets (10 packages loaded, 13248 targets configured).                                                                                                                      
INFO: Found 54 targets...                                                                                                                                                                      
INFO: Elapsed time: 2.877s, Critical Path: 1.27s                                                                                                                                               
INFO: 0 processes.                                                                                                                                                                             
INFO: Build completed successfully, 1 total action                                                                                                                                             
INFO: Build option --platforms has changed, discarding analysis cache.                                                                                                                         
INFO: Analysed target //:build-virtctl (0 packages loaded, 8939 targets configured).                                                                                                           
INFO: Found 1 target...                                                                                                                                                                        
Target //:build-virtctl up-to-date:                                                                                                                                                            
  bazel-genfiles/virtctl-copier                                                                                                                                                                
INFO: Elapsed time: 0.936s, Critical Path: 0.10s                                                                                                                                               
INFO: 1 process: 1 processwrapper-sandbox.                                                                                                                                                     
INFO: Build completed successfully, 2 total actions                                                                                                                                            
INFO: Build completed successfully, 2 total actions                                                                                                                                            
INFO: Build option --platforms has changed, discarding analysis cache.                                                                                                                         
INFO: Analysed target //:build-virtctl (0 packages loaded, 8939 targets configured).                                                                                                           
INFO: Found 1 target...                                                                                                                                                                        
Target //:build-virtctl up-to-date:                                                                                                                                                            
  bazel-genfiles/virtctl-copier                                                                                                                                                                
INFO: Elapsed time: 0.854s, Critical Path: 0.09s                                                                                                                                               
INFO: 0 processes.                                                                                                                                                                             
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
INFO: Build option --platforms has changed, discarding analysis cache.
INFO: Analysed target //:build-virtctl (0 packages loaded, 8941 targets configured).
INFO: Found 1 target...
Target //:build-virtctl up-to-date:
  bazel-genfiles/virtctl-copier
INFO: Elapsed time: 0.906s, Critical Path: 0.11s
INFO: 1 process: 1 processwrapper-sandbox.
INFO: Build completed successfully, 2 total actions
INFO: Build completed successfully, 2 total actions
INFO: Build option --platforms has changed, discarding analysis cache.
INFO: Analysed target //:build-virtctl (0 packages loaded, 8307 targets configured).
INFO: Found 1 target...
Target //:build-virtctl up-to-date:
  bazel-genfiles/virtctl-copier
INFO: Elapsed time: 0.723s, Critical Path: 0.11s
INFO: 1 process: 1 processwrapper-sandbox.
INFO: Build completed successfully, 2 total actions
INFO: Build completed successfully, 2 total actions
[+] Building 0.1s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                      0.0s
 => => transferring dockerfile: 38B                                                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                 
 => [internal] load metadata for docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12                                                       0.0s
 => [1/4] FROM docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12                                                                         0.0s
 => => resolve docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12                                                                         0.0s
 => [internal] load build context                                                                                                                                                         0.0s
 => => transferring context: 33B                                                                                                                                                          0.0s
 => CACHED [2/4] RUN     mkdir -p /go &&     source /etc/profile.d/gimme.sh &&     go get github.com/mattn/goveralls &&     go get -u github.com/golang/mock/gomock &&     go get -u git  0.0s
 => CACHED [3/4] RUN pip install j2cli && pip3 install operator-courier==1.3.0                                                                                                            0.0s
 => CACHED [4/4] COPY rsyncd.conf /etc/rsyncd.conf                                                                                                                                        0.0s
 => exporting to image                                                                                                                                                                    0.0s
 => => exporting layers                                                                                                                                                                   0.0s
 => => writing image sha256:6e512c03eb7b69fd0cca43aa2266799c72fead3ffa535c0390cfd7ae493d0ac8                                                                                              0.0s
 => => naming to docker.io/library/kubevirt                                                                                                                                               0.0s
go version go1.11.5 linux/amd64

go version go1.11.5 linux/amd64
devel devel_alt
devel
INFO: Build options --define and --platforms have changed, discarding analysis cache.
INFO: Analysed target //:push-images (0 packages loaded, 13125 targets configured).
INFO: Found 1 target...
Target //:push-images up-to-date:
  bazel-bin/push-images
INFO: Elapsed time: 1.678s, Critical Path: 0.32s
INFO: 0 processes.
INFO: Build completed successfully, 20 total actions
INFO: Build completed successfully, 20 total actions
F0726 08:20:35.171421   24317 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/virt-handler:devel: (503, 'Service Unavailable')
F0726 08:20:35.173954   24319 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/virt-operator:devel: (503, 'Service Unavailable')
F0726 08:20:35.174829   24315 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/virt-launcher:devel: (503, 'Service Unavailable')
F0726 08:20:35.226880   24312 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/example-cloudinit-hook-sidecar:devel: (503, 'Service Unavailable')
F0726 08:20:35.227720   24304 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/example-hook-sidecar:devel: (503, 'Service Unavailable')
F0726 08:20:35.228996   24308 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/subresource-access-test:devel: (503, 'Service Unavailable')
F0726 08:20:35.230473   24324 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/vm-killer:devel: (503, 'Service Unavailable')
F0726 08:20:35.234380   24306 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/container-disk-v1alpha:devel: (503, 'Service Unavailable')
F0726 08:20:35.246773   24321 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/cirros-container-disk-demo:devel: (503, 'Service Unavailable')
F0726 08:20:35.264621   24318 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/virt-api:devel: (503, 'Service Unavailable')
F0726 08:20:35.286813   24320 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/alpine-container-disk-demo:devel: (503, 'Service Unavailable')
F0726 08:20:35.289851   24325 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/winrmcli:devel: (503, 'Service Unavailable')
F0726 08:20:35.299104   24330 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/fedora30-cloud-container-disk-demo:devel: (503, 'Service Unavailable')
F0726 08:20:35.305036   24323 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/fedora-cloud-container-disk-demo:devel: (503, 'Service Unavailable')
F0726 08:20:35.305849   24329 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/cdi-http-import-server:devel: (503, 'Service Unavailable')
.
.
. <snip>
.                                    

Environment:

  • KubeVirt version (use virtctl version):
  • Kubernetes version (use kubectl version):
  • VM or VMI specifications:
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:
Master git commit: f680f4122bfa8da5bfb27cba8a51d5692e2df8b6
$ go version
go version go1.12.4 linux/amd64

updates

OS : Linux HP-workstation 4.15.0-54-generic #58-Ubuntu
Docker version:  19.03.1
KUBEVIRT_PROVIDER = k8s-1.13.3

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 24 (24 by maintainers)

Most upvoted comments

@kunalkushwaha thanks for sharing the how-to solve this issue. And I am glad that this issue is sorted out.

glad that you sorted this out! 🙂

@kunalkushwaha Give me some time will check it locally.

nope it is not, currently we are building via bazel, so all images exist under the bazel cache and not under the local registry

even after make clean, make bazel-build-images skips building images and push to local registry.

That the images aren’t rebuild after a clean is strange indeed. That no push happens is expected, because bazel-build-images does not push at all. That’s done by bazel-push-images

The bazel server is running in docker, and all cached files are in a docker volume. You can try do delete the latter and retry:

$ docker kill kubevirt-bazel-server && docker rm kubevirt-bazel-server
$ docker volume rm kubevirt