che: Push SHA tags for each nightly/next tag (was Start nightly Che workspaces is broken)

Describe the bug

Unable to start new workspace.

Che version

latest
nightly
other: please specify

Steps to reproduce

Create new workspace using chectl.
Try to start new workspace.
See that workspace fails to start with error:

Failed to pull image "quay.io/eclipse/che-theia-endpoint-runtime-binary@sha256:f07e6...ba2": 
rpc error: code = Unknown desc = Error response from daemon: manifest for 
quay.io/eclipse/che-theia-endpoint-runtime-binary@sha256:f0....a2 not found: 
manifest unknown: manifest unknown'

Expected behavior

Workspace should be executed.

Runtime

kubernetes (include output of kubectl version)
Openshift (include output of oc version)
minikube (include output of minikube version and kubectl version)
minishift (include output of minishift version and oc version)
docker-desktop + K8S (include output of docker version and kubectl version)
other: (please specify)

Screenshots

Installation method

chectl
- provide a full command that was used to deploy Eclipse Che (including the output)
- provide an output of chectl version command
OperatorHub
I don’t know

Environment

my computer
- Windows
- Linux
- macOS
Cloud
- Amazon
- Azure
- GCE
- other (please specify)
other: please specify

Eclipse Che Logs

unknown, workspaceu5fo1hivysjwxhfe.cpp-dev-7845cb785b-ncnsw
2021-02-01 07:49:17,507[aceSharedPool-0]  [WARN ] [.i.k.KubernetesInternalRuntime 257]  - Failed to start Kubernetes runtime of workspace workspaceu5fo1hivysjwxhfe.
org.eclipse.che.api.workspace.server.spi.InfrastructureException: Unrecoverable event occurred: 'Failed', 'Failed to pull image "quay.io/eclipse/che-theia-endpoint-runtime-binary@sha256:f07e63102eff29069524734d10dab480e34fdace006c284e6e91c5bbb3104ba2": rpc error: code = Unknown desc = Error response from daemon: manifest for quay.io/eclipse/che-theia-endpoint-runtime-binary@sha256:f07e63102eff29069524734d10dab480e34fdace006c284e6e91c5bbb3104ba2 not found: manifest unknown: manifest unknown', 'workspaceu5fo1hivysjwxhfe.cpp-dev-7845cb785b-ncnsw'
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.handleUnrecoverableEvent(KubernetesInternalRuntime.java:1019)
	at org.eclipse.che.workspace.infrastructure.kubernetes.util.UnrecoverablePodEventListener.handle(UnrecoverablePodEventListener.java:44)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.lambda$eventReceived$0(KubernetesDeployments.java:585)
	at java.base/java.util.concurrent.ConcurrentLinkedQueue.forEachFrom(Unknown Source)
	at java.base/java.util.concurrent.ConcurrentLinkedQueue.forEach(Unknown Source)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:585)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:550)
	at io.fabric8.kubernetes.client.utils.WatcherToggle.eventReceived(WatcherToggle.java:49)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:237)
	at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
	at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
	at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
	at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
2021-02-01 07:49:18,151[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 995]   - Workspace 'admin:cpp-w4qb6' with id 'workspaceu5fo1hivysjwxhfe' start failed
Logs from Feb 1,

Additional context

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16 (16 by maintainers)

Most upvoted comments

So, I should get the new registry after I executed kc rollout restart deployment plugin-registry but I don’t. I face this issue for a long time, is there is no changes in pod configuration, new pod is not run even if new container is available. Seems we need a dedicated issue for that on operator side (maybe it’s actual for helm chart as well). If it’s not expected behavior which would be strange.

To workaround this issue I should

kubectl patch deployment/plugin-registry \
        --patch "{\"spec\":{\"replicas\":0}}
kubectl patch deployment/plugin-registry \
        --patch "{\"spec\":{\"replicas\":1}}

and the issue with broken che-theia-endpoint-runtime-binary disappeared after that.

sleshchenko on Feb 1, 2021